ROS2 FFMPeg encoder/decoder

This ROS2 package supports encoding/decoding with the FFMpeg library, for example encoding h264 and h265 or HEVC, using Nvidia or other hardware acceleration when available. This package is meant to be used by image transport plugins like the ffmpeg image transport and the foxglove compressed video transport.

Supported systems

Continuous integration is tested under Ubuntu with the following ROS2 distros:

Installation

From packages

sudo apt-get install ros-${ROS_DISTRO}-ffmpeg-encoder-decoder

From source

Set the following shell variables:

repo=ffmpeg_encoder_decoder
url=https://github.com/ros-misc-utilities/${repo}.git

and follow the instructions here

Make sure to source your workspace’s install/setup.bash afterwards.

API overview

Preliminaries

When using libav it is important to understand the difference between the encoder and the codec. The codec is the standardized format in which images are encoded, for instance h264 or hevc. The encoder is a libav software module that can encode images for a given codec. For instance libx264, libx264rgb, h264_nvenc, and h264_vaapi` are all encoders that encode for the codec h264. Some of the encoders are hardware accelerated, some can handle more image formats than others, but in the end they all encode video for a specific codec.

For the many AV options available for various libav encoders, and for qmax, bit_rate and similar settings please refer to the ffmpeg documentation.

The inner workings

The diagram above shows the stations that a ROS Image message passes as it traverses the encoder and decoder. 1) The ROS image (with ROS sensor_msgs/Image encoding) is first converted with the ROS cv_bridge into the cv_bridge_target_format. This conversion is necessary because some ROS encodings (like bayer images) are not supported by libswscale. The cv_bridge_target_format can be set via setCVBridgeTargetFormat(const std::string & fmt). If this format is not set explicitly the image will be converted to the default format of bgr8. This may not be what you want for e.g. mono8 (gray) or Bayer images. Ideally thecv_bridge_target_format can be directly used by the libav decoder so the next step becomes a no-op. But at the very least cv_bridge_target_format must be an acceptable libswscale input format (with the exception of special hacks for encoding single-channel images, see below). 2) The image is then converted to av_source_pixel_format using libswscale. The av_source_pixel_format can be set with setAVSourcePixelFormat(), defaulting to something that is acceptable to the libav encoder. You can use ffmpeg (ffmpeg -h encoder=libx264 | grep formats) to list all formats that an encoder supports. Note that the ffmpeg/libav format string notation is different from the ROS encoding strings, and the av_source_pixel_format is specified using the libav convention, whereas the cv_bridge_target_format uses ROS convention! (If you choose to bypass the cv_bridge conversion from step 1 by feeding the images to the encoder directly via the encodeImage(const cv::Mat & img ...) method, you must still set the cv_bridge_target_format such that the encoder knows what format the img argument has.) When aiming for lossless compression, beware of any av_source_pixel_format that reduces the color resolution, such as yuv420p, nv12 etc. For Bayer images, use the special hack for single-channel images.

3) The libav encoder encodes the packet with its supported codec, e.g. the libx264 will produce h264 packets. The encoding field of the FFMPEGPacket message will document all image format conversions and the codec, in reverse order, separated by semicolon. This way the decoder can attempt to reproduce the original ros_encoding.

4) The libav decoder decodes the packet into the original av_source_pixel_format.

5) Finally the image is converted to output_message_format using libswscale. This format can be set (in ROS encoding syntax!) with setOutputMessageEncoding(). The format must be supported by both ROS and libswscale (except when using the special hack for single-channel images).

Note that only very few combinations of libav encoders, cv_bridge_target_format and av_source_pixel_format have been tested. Please provide feedback if you observe crashes or find obvious bugs. PRs are always appreciated!

The special single-channel hack

Many libav encoders do not support single-channel formats (like mono8 or bayer). For this reason a special hack is implemented in the encoder that adds an empty (zero-value) color channel to the single-channel image. Later, the decoder removes it again. To utilitze this hack, specify a cv_bridge_target_format of e.g. bayer_rggb8. Without the special hack, this would trigger an error because Bayer formats are not acceptable to libswscale. Instead, the image is converted to yuv420p or nv12 by adding an empty color channel. These formats are acceptable to most encoders. The decoder in turn recognizes that the cv_bridge_target_format is a single-channel format, but yuv420p/nv12 are not, and therefore drops the color channel. This hack greatly improves the efficiency for lossless encoding of Bayer images because it avoids conversion to full RGB and back.

API usage

Encoder

Using the encoder involves the following steps:

• instantiating the Encoder object.
• setting properties like the libav encoder to use, the encoding formats, and AV options.
• initializing the encoder object. This requires knowledge of the image size and therefore

File truncated at 100 lines see the full file

Any contribution that you make to this repository will be under the Apache 2 License, as dictated by that license:

5. Submission of Contributions. Unless You explicitly state otherwise,
   any Contribution intentionally submitted for inclusion in the Work
   by You to the Licensor shall be under the terms and conditions of
   this License, without any additional terms or conditions.
   Notwithstanding the above, nothing herein shall supersede or modify
   the terms of any separate license agreement you may have executed
   with Licensor regarding such Contributions.

Contributors must sign-off each commit by adding a Signed-off-by: ... line to commit messages to certify that they have the right to submit the code they are contributing to the project according to the Developer Certificate of Origin (DCO).