Package Summary

Tags No category tags.
Version 0.6.0
License Apache-2.0
Build type AMENT_CMAKE
Use RECOMMENDED

Repository Summary

Checkout URI https://github.com/ros2/rosbag2.git
VCS Type git
VCS Version humble
Last Updated 2022-12-29
Dev Status DEVELOPED
CI status No Continuous Integration
Released RELEASED
Tags No category tags.
Contributing Help Wanted (0)
Good First Issues (0)
Pull Requests to Review (0)

Package Description

rosbag2 storage plugin using the MCAP file format

Additional Links

No additional links.

Maintainers

  • Foxglove
  • ROS Tooling Working Group

Authors

No additional authors.

rosbag2_storage_mcap

This package provides a storage plugin for rosbag2 which extends it with support for the MCAP file format.

ROS Foxy version ROS Galactic version ROS Humble version ROS Rolling version

⚠️ Move from ros-tooling organization

This package was recently merged into https://github.com/ros2/rosbag2 from https://github.com/ros-tooling/rosbag2_storage_mcap. To view historical pull requests and git history of this package, check here.

Installation

rosbag2_storage_mcap is available as part of the current ROS 2 distributions. On Ubuntu, after following the ROS 2 installation instructions, you can use:

# Replace "rolling" with your ROS distro (`echo $ROS_DISTRO`)
$ sudo apt install ros-rolling-rosbag2-storage-mcap

Usage

Use MCAP files with regular ros2 bag commands by adding the --storage mcap option (abbreviated as -s mcap):

$ ros2 bag record -s mcap /topic1 /topic2 ...

$ ros2 bag play -s mcap path/to/your_recording.mcap

$ ros2 bag info -s mcap path/to/your_recording.mcap

Writer Configuration

To configure details of the MCAP writer for ros2 bag record, use the --storage-config-file options to provide a YAML file describing mcap::McapWriterOptions. Field descriptions below copied from McapWriterOptions declaration

Field Type / Values Description
noChunkCRC bool Disable CRC calculation for Chunks. Ignored if noChunking=true.
noAttachmentCRC bool Disable CRC calculation for Attachments.
enableDataCRC bool Enables CRC calculation for the entire Data section. Useful when noChunking=True.
noSummaryCRC bool Disable CRC calculation for the Summary section.
noChunking bool Do not write Chunks to the file, instead writing Schema, Channel, and Message records directly into the Data section.
noMessageIndex bool Do not write Message Index records to the file. If noSummary=true and noChunkIndex=false, Chunk Index records will still be written to the Summary section, providing a coarse message index.
noSummary bool Do not write Summary or Summary Offset sections to the file, placing the Footer record immediately after DataEnd. This can provide some speed boost to file writing and produce smaller files, at the expense of requiring a conversion process later if fast summarization or indexed access is desired.
chunkSize unsigned int Target uncompressed Chunk payload size in bytes. Once a Chunk's uncompressed data meets or exceeds this size, the Chunk will be compressed (if compression is enabled) and written to disk. Note that smaller Chunks may be written, such as the last Chunk in the Data section. This option is ignored if noChunking=true.
compression "None", "Lz4", "Zstd" Compression algorithm to use when writing Chunks. This option is ignored if noChunking=true.
compressionLevel "Fastest", "Fast", "Default", "Slow", "Slowest" Compression level to use when writing Chunks. Slower generally produces smaller files, at the expense of more CPU time. These levels map to different internal settings for each compression algorithm.
forceCompression bool By default, Chunks that do not benefit from compression will be written uncompressed. This option can be used to force compression on all Chunks. This option is ignored if noChunking=true.
noRepeatedSchemas bool Advanced option.
noRepeatedChannels bool Advanced option.
noMetadataIndex bool Advanced option.
noChunkIndex bool Advanced option.
noStatistics bool Advanced option.
noSummaryOffsets bool Advanced option.

Example:

# mcap_writer_options.yml
noChunkCRC: false
noChunking: false
noMessageIndex: false
noSummary: false
chunkSize: 786432
compression: "Zstd"
compressionLevel: "Fast"
forceCompression: false

$ ros2 bag record -s mcap -o my_bag --all --storage-config-file mcap_writer_options.yml

Storage Preset Profiles

You can also use one of the preset profiles described below, for example:

$ ros2 bag record -s mcap -o my_bag --all --storage-preset-profile fastwrite

fastwrite

Configures the MCAP writer for the highest possible write throughput and lowest resource utilization. This preset does not calculate CRCs for integrity checking, and does not write a message index. This preset profile is useful for resource-constrained robots.

Equivalent to this storage configuration:

noChunking: true
noSummaryCRC: true

Using MCAPs written with fastwrite as a long-term storage format is not recommended. Some features will not work when reading MCAP files without a message index, such as reading messages from a subset of topics or seeking. When recording MCAPs on your robot with fastwrite, it is a good idea to post-process these files afterwards, to restore the message index and also save storage space:

# Using the MCAP CLI https://github.com/foxglove/mcap/tree/main/go/cli/mcap
$ mcap compress fast.mcap -o compressed.mcap
# Using `ros2 bag convert`
$ cat << EOF > convert.yaml
output_bags:
  - uri: compressed
    storage_id: mcap
    storage_preset_profile: zstd_small
EOF
$ ros2 bag convert -i fast.mcap -o convert.yaml

Equivalent to this storage configuration:

noChunking: true
noSummaryCRC: true

zstd_fast

Configures the MCAP writer to use chunk compression with zstd. Chunk compression yields file sizes comparable to bags compressed with file-level compression, but allows tools to efficiently read messages without decompressing the entire bag. This preset uses the lowest compression ratio and disables CRC calculation, to achieve high throughput while conserving disk space.

Equivalent to this storage configuration:

compression: "Zstd"
compressionLevel: "Fastest"
noChunkCRC: true

zstd_small

Configures the MCAP writer to write 4MB chunks, compressed with zstd using its highest compression ratio. This produces very small bags, but can be resource-intensive to write. This preset also calculates chunk CRCs, which allow a reader to determine if a chunk is corrupted. This preset is useful when using ros2 bag convert as a post-processing step.

Equivalent to this storage configuration:

compression: "Zstd"
compressionLevel: "Slowest"
chunkSize: 4194304 # 4 * 1024 * 1024

ROS 2 Distro maintenance

Whenever a ROS 2 distribution reaches EOL, search for comments marked COMPATIBILITY - which may no longer be needed when no new releases will be made for that distro.

CHANGELOG

Changelog for package rosbag2_storage_mcap

0.6.0 (2022-11-28)

  • mcap_storage: \'none\' is a valid storage preset profile (#86)
  • mcap_storage: handle update_metadata call (#83)
  • Update clang-format rules to fit ROS 2 style guide (#80)
  • Revert \"read_order: throw exception from set_read_order for unsupported orders\" This reverts commit aef9b9a65293f9e5d80a858ef84e485a8655a0c0.
  • read_order: throw exception from set_read_order for unsupported orders
  • Fix compile flags to work on rosbag_storage:0.17.x (#78) This fixes the compile flags for rolling, which has two versions -- one that does not support read order (0.17.x) and one that does support read order (0.18.x).
  • Fix Windows build (#73) Update mcap version to newest windows-compatible release. Add visibility macros for tests. Add clang-format preprocessor indentation for visibility_control to be readable.
  • Contributors: Andrew Symington, Emerson Knapp, James Smith, james-rms

0.5.0 (2022-11-02)

  • set defaults for SQLite plugin parity (#68)
  • rosbag2_storage_mcap: add storage preset profiles (#57)
  • rename test_fixture_interfaces package to testdata (#64)
  • Switch to using the vendored zstd library. (#59)
  • Add set_read_order reader API (#54)
  • Contributors: Chris Lalancette, Emerson Knapp, James Smith

0.4.0 (2022-10-06)

  • Some minor improvements in rosbag2_storage_mcap after review (#58)

    1. Fixed some findings from Clang-Tidy
    2. Some renames according to the ROS2 coding style
    3. Add default initializations for member variables

    1. Moved code responsible for adding schema and channel from write(msg) to create_topic(topic) method to reduce performance burden on first message write and in lieu to preparation for moving schema collection process to upper SequentialWriter layer.

  • Revert \"rosbag2_storage_mcap: add storage preset profiles\" This reverts commit 38830add3935b978968fe2703d3180b413ccc8c2.

  • rosbag2_storage_mcap: add storage preset profiles

  • Contributors: James Smith, Michael Orlov

0.3.0 (2022-09-09)

  • Store IDL message definitions in Schema records when no MSG definition is available (#43)
  • Contributors: James Smith

0.2.0 (2022-09-08)

  • Support timestamp-ordered playback (#50)
  • Support regex topic filtering
  • Contributors: James Smith

0.1.7 (2022-08-15)

  • Add all lz4 sources to fix undefined symbols at runtime (#46)
  • Contributors: Emerson Knapp

0.1.6 (2022-07-22)

0.1.5 (2022-04-25)

  • Fix build for Foxy (#34)
  • Contributors: Jacob Bandes-Storch

0.1.4 (2022-04-21)

  • fix: minor issues (#31)
    • remove unnecessary block
    • use target_link_libraries instead of ament_target_dependencies
    • remove ros environment
    • add prefix to compile definition
  • Update email address for Foxglove maintainers (#32)
  • Contributors: Daisuke Nishimatsu, Jacob Bandes-Storch

0.1.3 (2022-04-20)

0.1.2 (2022-04-20)

  • Added mcap_vendor package. Updated CMakeLists.txt to fetch dependencies with FetchContent rather than Conan.
  • Contributors: Jacob Bandes-Storch

0.1.1 (2022-04-01)

  • CMake build script will now execute pip install conan automatically.
  • Contributors: Daisuke Nishimatsu

0.1.0 (2022-03-24)

  • [1.0.0] Use Summary section for get_metadata() and seek(), implement remaining methods (#17)
  • feat: add play impl (#16)
  • chore: refine package.xml (#15)
  • Don\'t throw when READ_WRITE mode is used; add .mcap file extension to recorded files (#14) I may be missing something, but from a cursory glance at [this code](https://github.com/ros2/rosbag2/blob/342d8ed3c1c4ae0411a4a92b60e79a728b8974b8/rosbag2_storage/src/rosbag2_storage/impl/storage_factory_impl.hpp#L108-L135), it appears that the APPEND mode is never used. This means we need to support READ_WRITE. This also adds a .mcap extension to recorded file names.
  • Add dynamic message definition lookup (#13) Currently, an exception will be thrown if lookup fails.
  • Switch C++ formatter to clang-format (#12) Remove uncrustify linter in favor of clang-format, which is easier to configure for use in VS Code format-on-save.
  • Merge pull request #7 from ros-tooling/jhurliman/reader-writer Reader and writer implementation
  • uninitialized struct
  • lint
  • lint
  • lint
  • Reader and writer implementation
  • Merge pull request #6 from wep21/add-metadata-impl feat: add metadata impl
  • feat: add metadata impl
  • Merge pull request #5 from wep21/mcap-storage-impl feat: mcap storage impl
  • chore: update cmake minimum version
  • chore: install mcap header
  • chore: include mcap header
  • fix: move fetch content into rosbag2 storage mcap
  • Merge pull request #3 from ros-tooling/emersonknapp/mcap_plugin_skeleton Add mcap storage plugin skeleton and CI
  • Add rosbag2_storage_mcap skeleton
  • Contributors: Daisuke Nishimatsu, Emerson Knapp, Jacob Bandes-Storch, John Hurliman, wep21

Wiki Tutorials

See ROS Wiki Tutorials for more details.

Source Tutorials

Not currently indexed.

Launch files

No launch files found

Messages

No message files found.

Services

No service files found

Plugins

No plugins found.

Recent questions tagged rosbag2_storage_mcap at answers.ros.org

Package Summary

Tags No category tags.
Version 0.6.0
License Apache-2.0
Build type AMENT_CMAKE
Use RECOMMENDED

Repository Summary

Checkout URI https://github.com/ros2/rosbag2.git
VCS Type git
VCS Version rolling
Last Updated 2022-12-28
Dev Status DEVELOPED
CI status No Continuous Integration
Released RELEASED
Tags No category tags.
Contributing Help Wanted (0)
Good First Issues (0)
Pull Requests to Review (0)

Package Description

rosbag2 storage plugin using the MCAP file format

Additional Links

No additional links.

Maintainers

  • Foxglove
  • Michael Orlov
  • ROS Tooling Working Group

Authors

No additional authors.

rosbag2_storage_mcap

This package provides a storage plugin for rosbag2 which extends it with support for the MCAP file format.

ROS Foxy version ROS Galactic version ROS Humble version ROS Rolling version

⚠️ Move from ros-tooling organization

This package was recently merged into https://github.com/ros2/rosbag2 from https://github.com/ros-tooling/rosbag2_storage_mcap. To view historical pull requests and git history of this package, check here.

Installation

rosbag2_storage_mcap is available as part of the current ROS 2 distributions. On Ubuntu, after following the ROS 2 installation instructions, you can use:

# Replace "rolling" with your ROS distro (`echo $ROS_DISTRO`)
$ sudo apt install ros-rolling-rosbag2-storage-mcap

Usage

Use MCAP files with regular ros2 bag commands by adding the --storage mcap option (abbreviated as -s mcap):

$ ros2 bag record -s mcap /topic1 /topic2 ...

$ ros2 bag play -s mcap path/to/your_recording.mcap

$ ros2 bag info -s mcap path/to/your_recording.mcap

Writer Configuration

To configure details of the MCAP writer for ros2 bag record, use the --storage-config-file options to provide a YAML file describing mcap::McapWriterOptions. Field descriptions below copied from McapWriterOptions declaration

Field Type / Values Description
noChunkCRC bool Disable CRC calculation for Chunks. Ignored if noChunking=true.
noAttachmentCRC bool Disable CRC calculation for Attachments.
enableDataCRC bool Enables CRC calculation for the entire Data section. Useful when noChunking=True.
noSummaryCRC bool Disable CRC calculation for the Summary section.
noChunking bool Do not write Chunks to the file, instead writing Schema, Channel, and Message records directly into the Data section.
noMessageIndex bool Do not write Message Index records to the file. If noSummary=true and noChunkIndex=false, Chunk Index records will still be written to the Summary section, providing a coarse message index.
noSummary bool Do not write Summary or Summary Offset sections to the file, placing the Footer record immediately after DataEnd. This can provide some speed boost to file writing and produce smaller files, at the expense of requiring a conversion process later if fast summarization or indexed access is desired.
chunkSize unsigned int Target uncompressed Chunk payload size in bytes. Once a Chunk's uncompressed data meets or exceeds this size, the Chunk will be compressed (if compression is enabled) and written to disk. Note that smaller Chunks may be written, such as the last Chunk in the Data section. This option is ignored if noChunking=true.
compression "None", "Lz4", "Zstd" Compression algorithm to use when writing Chunks. This option is ignored if noChunking=true.
compressionLevel "Fastest", "Fast", "Default", "Slow", "Slowest" Compression level to use when writing Chunks. Slower generally produces smaller files, at the expense of more CPU time. These levels map to different internal settings for each compression algorithm.
forceCompression bool By default, Chunks that do not benefit from compression will be written uncompressed. This option can be used to force compression on all Chunks. This option is ignored if noChunking=true.
noRepeatedSchemas bool Advanced option.
noRepeatedChannels bool Advanced option.
noMetadataIndex bool Advanced option.
noChunkIndex bool Advanced option.
noStatistics bool Advanced option.
noSummaryOffsets bool Advanced option.

Example:

# mcap_writer_options.yml
noChunkCRC: false
noChunking: false
noMessageIndex: false
noSummary: false
chunkSize: 786432
compression: "Zstd"
compressionLevel: "Fast"
forceCompression: false

$ ros2 bag record -s mcap -o my_bag --all --storage-config-file mcap_writer_options.yml

Storage Preset Profiles

You can also use one of the preset profiles described below, for example:

$ ros2 bag record -s mcap -o my_bag --all --storage-preset-profile fastwrite

fastwrite

Configures the MCAP writer for the highest possible write throughput and lowest resource utilization. This preset does not calculate CRCs for integrity checking, and does not write a message index. This preset profile is useful for resource-constrained robots.

Equivalent to this storage configuration:

noChunking: true
noSummaryCRC: true

Using MCAPs written with fastwrite as a long-term storage format is not recommended. Some features will not work when reading MCAP files without a message index, such as reading messages from a subset of topics or seeking. When recording MCAPs on your robot with fastwrite, it is a good idea to post-process these files afterwards, to restore the message index and also save storage space:

# Using the MCAP CLI https://github.com/foxglove/mcap/tree/main/go/cli/mcap
$ mcap compress fast.mcap -o compressed.mcap
# Using `ros2 bag convert`
$ cat << EOF > convert.yaml
output_bags:
  - uri: compressed
    storage_id: mcap
    storage_preset_profile: zstd_small
EOF
$ ros2 bag convert -i fast.mcap -o convert.yaml

Equivalent to this storage configuration:

noChunking: true
noSummaryCRC: true

zstd_fast

Configures the MCAP writer to use chunk compression with zstd. Chunk compression yields file sizes comparable to bags compressed with file-level compression, but allows tools to efficiently read messages without decompressing the entire bag. This preset uses the lowest compression ratio and disables CRC calculation, to achieve high throughput while conserving disk space.

Equivalent to this storage configuration:

compression: "Zstd"
compressionLevel: "Fastest"
noChunkCRC: true

zstd_small

Configures the MCAP writer to write 4MB chunks, compressed with zstd using its highest compression ratio. This produces very small bags, but can be resource-intensive to write. This preset also calculates chunk CRCs, which allow a reader to determine if a chunk is corrupted. This preset is useful when using ros2 bag convert as a post-processing step.

Equivalent to this storage configuration:

compression: "Zstd"
compressionLevel: "Slowest"
chunkSize: 4194304 # 4 * 1024 * 1024

ROS 2 Distro maintenance

Whenever a ROS 2 distribution reaches EOL, search for comments marked COMPATIBILITY - which may no longer be needed when no new releases will be made for that distro.

CHANGELOG

Changelog for package rosbag2_storage_mcap

0.6.0 (2022-11-28)

  • mcap_storage: \'none\' is a valid storage preset profile (#86)
  • mcap_storage: handle update_metadata call (#83)
  • Update clang-format rules to fit ROS 2 style guide (#80)
  • Revert \"read_order: throw exception from set_read_order for unsupported orders\" This reverts commit aef9b9a65293f9e5d80a858ef84e485a8655a0c0.
  • read_order: throw exception from set_read_order for unsupported orders
  • Fix compile flags to work on rosbag_storage:0.17.x (#78) This fixes the compile flags for rolling, which has two versions -- one that does not support read order (0.17.x) and one that does support read order (0.18.x).
  • Fix Windows build (#73) Update mcap version to newest windows-compatible release. Add visibility macros for tests. Add clang-format preprocessor indentation for visibility_control to be readable.
  • Contributors: Andrew Symington, Emerson Knapp, James Smith, james-rms

0.5.0 (2022-11-02)

  • set defaults for SQLite plugin parity (#68)
  • rosbag2_storage_mcap: add storage preset profiles (#57)
  • rename test_fixture_interfaces package to testdata (#64)
  • Switch to using the vendored zstd library. (#59)
  • Add set_read_order reader API (#54)
  • Contributors: Chris Lalancette, Emerson Knapp, James Smith

0.4.0 (2022-10-06)

  • Some minor improvements in rosbag2_storage_mcap after review (#58)

    1. Fixed some findings from Clang-Tidy
    2. Some renames according to the ROS2 coding style
    3. Add default initializations for member variables

    1. Moved code responsible for adding schema and channel from write(msg) to create_topic(topic) method to reduce performance burden on first message write and in lieu to preparation for moving schema collection process to upper SequentialWriter layer.

  • Revert \"rosbag2_storage_mcap: add storage preset profiles\" This reverts commit 38830add3935b978968fe2703d3180b413ccc8c2.

  • rosbag2_storage_mcap: add storage preset profiles

  • Contributors: James Smith, Michael Orlov

0.3.0 (2022-09-09)

  • Store IDL message definitions in Schema records when no MSG definition is available (#43)
  • Contributors: James Smith

0.2.0 (2022-09-08)

  • Support timestamp-ordered playback (#50)
  • Support regex topic filtering
  • Contributors: James Smith

0.1.7 (2022-08-15)

  • Add all lz4 sources to fix undefined symbols at runtime (#46)
  • Contributors: Emerson Knapp

0.1.6 (2022-07-22)

0.1.5 (2022-04-25)

  • Fix build for Foxy (#34)
  • Contributors: Jacob Bandes-Storch

0.1.4 (2022-04-21)

  • fix: minor issues (#31)
    • remove unnecessary block
    • use target_link_libraries instead of ament_target_dependencies
    • remove ros environment
    • add prefix to compile definition
  • Update email address for Foxglove maintainers (#32)
  • Contributors: Daisuke Nishimatsu, Jacob Bandes-Storch

0.1.3 (2022-04-20)

0.1.2 (2022-04-20)

  • Added mcap_vendor package. Updated CMakeLists.txt to fetch dependencies with FetchContent rather than Conan.
  • Contributors: Jacob Bandes-Storch

0.1.1 (2022-04-01)

  • CMake build script will now execute pip install conan automatically.
  • Contributors: Daisuke Nishimatsu

0.1.0 (2022-03-24)

  • [1.0.0] Use Summary section for get_metadata() and seek(), implement remaining methods (#17)
  • feat: add play impl (#16)
  • chore: refine package.xml (#15)
  • Don\'t throw when READ_WRITE mode is used; add .mcap file extension to recorded files (#14) I may be missing something, but from a cursory glance at [this code](https://github.com/ros2/rosbag2/blob/342d8ed3c1c4ae0411a4a92b60e79a728b8974b8/rosbag2_storage/src/rosbag2_storage/impl/storage_factory_impl.hpp#L108-L135), it appears that the APPEND mode is never used. This means we need to support READ_WRITE. This also adds a .mcap extension to recorded file names.
  • Add dynamic message definition lookup (#13) Currently, an exception will be thrown if lookup fails.
  • Switch C++ formatter to clang-format (#12) Remove uncrustify linter in favor of clang-format, which is easier to configure for use in VS Code format-on-save.
  • Merge pull request #7 from ros-tooling/jhurliman/reader-writer Reader and writer implementation
  • uninitialized struct
  • lint
  • lint
  • lint
  • Reader and writer implementation
  • Merge pull request #6 from wep21/add-metadata-impl feat: add metadata impl
  • feat: add metadata impl
  • Merge pull request #5 from wep21/mcap-storage-impl feat: mcap storage impl
  • chore: update cmake minimum version
  • chore: install mcap header
  • chore: include mcap header
  • fix: move fetch content into rosbag2 storage mcap
  • Merge pull request #3 from ros-tooling/emersonknapp/mcap_plugin_skeleton Add mcap storage plugin skeleton and CI
  • Add rosbag2_storage_mcap skeleton
  • Contributors: Daisuke Nishimatsu, Emerson Knapp, Jacob Bandes-Storch, John Hurliman, wep21

Wiki Tutorials

See ROS Wiki Tutorials for more details.

Source Tutorials

Not currently indexed.

Launch files

No launch files found

Messages

No message files found.

Services

No service files found

Plugins

No plugins found.

Recent questions tagged rosbag2_storage_mcap at answers.ros.org