![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged reductstore_agent at Robotics Stack Exchange
![]() |
reductstore_agent package from reductstore_agent reporeductstore_agent |
ROS Distro
|
Package Summary
Tags | No category tags. |
Version | 0.2.0 |
License | MIT |
Build type | AMENT_PYTHON |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/reductstore/reductstore_agent.git |
VCS Type | git |
VCS Version | main |
Last Updated | 2025-07-09 |
Dev Status | DEVELOPED |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Additional Links
Maintainers
- Anthony Cavin (
anthonycvn) - Alexey Timin (
atimin)
Authors
- Anthony Cavin
reductstore_agent
reductstore_agent is a ROS 2 node that records selected topics into ReductStore, a high-performance storage and streaming solution. ReductStore is an ELT-based system for robotics and industrial IoT data acquisition. It ingests and streams time-series data of any size—images, sensor readings, logs, files, MCAP, ROS bags—and stores it with time indexing and labels for ultra-fast retrieval and management.
This agent is fully configurable via YAML and designed to solve storage, bandwidth, and workflow limitations commonly found in field robotics. It streams data to ReductStore in near real-time with optional compression, splitting, dynamic labeling, and per-pipeline controls.
System Requirements
To use this agent, you must have a running instance of ReductStore. You can start a local instance using Docker, install it via Snap or from binaries. Refer to the official guide for setup instructions: ReductStore Getting Started Guide
This agent is tested with:
- ROS 2: Jazzy and Rolling
- OS: Ubuntu 24.04 (Noble)
- Python: 3.12
Motivation
- Continuous recording: Prevent oversized rosbag files by splitting recordings by time, size, or topic groups.
- Bandwidth constraints: Filter and compress data before optionally replicating to a central server or the cloud.
- Manual workflows: Replace manual drive swaps, custom scripts, and bag handling with automated data management.
- Lack of filtering: Apply dynamic labels (e.g., mission ID) to tag, search, and retrieve specific data segments.
- Ubuntu Core: Future Snap integration to support deployment as part of the Ubuntu Core observability stack.
Structure
The agent is configured using a YAML file. Each pipeline is an independent logging unit (only one type of pipeline is supported at the moment where all topics are recorded continuously without filtering).
/**/*:
ros__parameters:
storage: # local ReductStore instance
url: "http://localhost:8383"
api_token: "access_token"
bucket: "ros_data"
quota_type: "FIFO"
quota_size: "200GB"
pipelines:
telemetry:
filename_mode: "timestamp"
include_topics:
- "/camera/.*"
exclude_topics:
- "/camera/ignore"
static_labels:
source: telemetry
robot: alpha
split:
max_duration_s: 3600
max_size_bytes: 10000
See the Configuration section for details on available parameters.
Installing
Build and run in a ROS 2 workspace:
# 1. Clone your repo and enter the workspace
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
git clone https://github.com/reductstore/reductstore_agent.git
cd ..
# 2. Install system dependencies
rosdep install --from-paths src --ignore-src -r -y
# 3. Build your package
colcon build --packages-select reductstore_agent
# 4. Source the workspace and run your node
source install/local_setup.bash
ros2 run reductstore_agent recorder --ros-args --params-file ./config.yaml
Configuration
The configuration file is a YAML file that defines the storage settings and pipelines. The storage
section contains ReductStore connection details, and the pipelines
section defines the individual pipelines for recording data.
Storage Configuration
The storage
section specifies the ReductStore instance to connect to:
-
url
: The URL of the ReductStore instance (e.g.,http://localhost:8383
). -
api_token
: The API token for authentication. This is required to access the ReductStore instance. -
bucket
: The bucket name where the data will be stored. -
quota_type
: The type of quota to apply. Options are:-
"FIFO"
: First In, First Out (oldest data is removed first). -
"HARD"
: Hard limit (data is not accepted when the quota is reached). -
"NONE"
: No quota applied.
-
-
quota_size
: The size of the quota in bytes. This is required ifquota_type
is set to"FIFO"
or"HARD"
. -
max_block_size
: Maximum size of each block in bytes. -
max_block_records
: Maximum number of records per block.
More information on how to setup ReductStore can be found in the ReductStore Getting Started Guide.
Pipeline Parameters
File truncated at 100 lines see the full file
Changelog for package reductstore_agent
0.2.0 (2025-07-09)
- Merge pull request #23 from reductstore/9-add-support-for-include_regex-and-exclude_regex Add support for include/exclude topics using regex
- Merge pull request #24 from reductstore/12-add-support-for-static-labels-from-pipeline-config Add support for static labels from pipeline config
- Merge pull request #22 from reductstore/8-add-reductstore-bucket-settings-to-config Add support for bucket settings
0.1.0 (2025-06-09)
- Merge pull request #20 from reductstore/16-open-pr-to-rosrosdistro-to-publish-ros-package Update name to reductstore_agent
- Merge pull request #19 from reductstore/7-enable-pylint-in-github-actions Add code linting with flake8, pep257, and copyright checks
- Merge pull request #6 from reductstore/3-minimum-recording-implementation Implement minimum recorder agent for ROS 2 MCAP files
Package Dependencies
Deps | Name |
---|---|
ament_copyright | |
ament_flake8 | |
ament_pep257 | |
rclpy | |
std_msgs | |
std_srvs |
System Dependencies
Name |
---|
python3-pytest |