Use quality-of-service settings to handle lossy networks¶
Table of Contents
Please read the documentationg page about QoS settings for background information on available support in ROS 2.
In this demo, we will spawn a node that publishes a camera image and another that subscribes to the image and shows it on the screen. We will then simulate a lossy network connection between them and show how different quality of service settings handle the bad link.
Simply download the binary packages for your OS from the installation page.
Before running the demo, make sure you have a working webcam connected to your computer.
Once you’ve installed ROS 2, if you’re on Linux, source your setup.bash file:
. <path to ROS 2 install space>/setup.bash
or if you’re on Windows cmd:
call <path to ROS 2 install space>/local_setup.bat
ros2 run image_tools showimage
Nothing will happen yet.
showimage is a subscriber node that is waiting for a publisher on the
Note: you have to close the
showimage process with
You can’t just close the window.
In a separate terminal, source the install file and run the publisher node:
ros2 run image_tools cam2image
This will publish an image from your webcam. In case you don’t have a camera attached to your computer, there is a commandline option which publishes predefined images.
ros2 run image_tools cam2image -b
In this window, you’ll see terminal output:
Publishing image #1 Publishing image #2 Publishing image #3 ...
A window will pop up with the title “view” showing your camera feed. In the first window, you’ll see output from the subscriber:
Received image #1 Received image #2 Received image #3 ...
Note for OS X users: If you these examples do not work or you receive an error like
ddsi_conn_write failed -1 then you’ll need to increase your system wide UDP packet size:
$ sudo sysctl -w net.inet.udp.recvspace=209715 $ sudo sysctl -w net.inet.udp.maxdgram=65500
These changes will not persist a reboot. If you want the changes to persist, add these lines to
/etc/sysctl.conf (create the file if it doesn’t exist already):
In one of your terminals, add a -h flag to the original command:
ros2 run image_tools showimage -- -h
You’ll see a list of the possible options you can pass to the demo.
-h: The help message.
There are two options for this policy: reliable or best effort.
Reliable means that values may be reset and the underlying DDS publisher might block, in order for messages to get delivered in order.
Best effort means that messages will get sent as is, and they may get dropped or lost without effecting the behavior of the publisher.
-k: History policy (the “k” stands for “keep”).
Determines how DDS buffers messages in the time between the user code that called
publish and the time when the message actually gets sent.
There are two options for history: KEEP_ALL and KEEP_LAST.
KEEP_ALL will buffer all messages before they get sent.
KEEP_LAST limits the number of buffered messages to a depth specified by the user.
-d: Queue depth.
Only used if the history policy is set to KEEP_LAST.
The queue depth determines the maximum number of not yet received messages that get buffered on the sender’s side before messages start getting dropped.
-t TOPIC: Topic to use.
The topic to use (default: image)
If you run
cam2image -h, you’ll see the same set of command line options and some additional ones:
-s: Toggle displaying the input camera stream.
If you run
cam2image -s by itself, you’ll see a camera window.
If you also run
showimage, you’ll see two camera windows.
-y: Set the size of the camera feed (x sets the width, y sets the height).
-b: Produce images of burgers rather than connecting to a camera
-f: Publish frequency in Hz. (default: 30)
The default quality of service settings are tuned for maximum reliability: the reliability policy is reliable, and the history policy is “keep all”.
It’s worth noting that both ends must have the same reliability settings for this to work. If the consumer requires the publisher to be reliable, DDS will not match them and there won’t be any exchange between them.
We won’t see much of a difference if we change the quality of service settings, since the publisher and subscriber are passing messages over inter-process communication, and messages are unlikely to get dropped if they are travelling within the same machine.
This section of the demo won’t work on RTI’s Connext DDS. When running multiple nodes in the same host, the RTI Connext implementation uses shared memory along with the loopback interface. Degrading the loopback interface throughput won’t affect shared memory, thus traffic between the two nodes won’t be affected. See RMW implementation code and RTI Connext Documentation for further reference.
This next section is Linux-specific.
However, for OS X and Windows you can achieve a similar effect with the utilities “Network Link Conditioner” (part of the xcode tool suite) and “Clumsy” (http://jagt.github.io/clumsy/index.html), respectively, but they will not be covered in this tutorial.
We are going to use the Linux network traffic control utility,
sudo tc qdisc add dev lo root netem loss 5%
This magical incantation will simulate 5% packet loss over the local loopback device.
If you use a higher resolution of the images (e.g.
-x 640 -y 480) you might want to try a lower packet loss rate (e.g.
Next we start the
showimage, and we’ll soon notice that both programs seem to have slowed down the rate at which images are transmitted.
This is caused by the behavior of the default QoS settings.
Enforcing reliability on a lossy channel means that the publisher (in this case,
cam2image) will resend the network packets until it receives acknowledgement from the consumer (i.e.
Let’s now try running both programs, but with more suitable settings.
First of all, we’ll use the
-r 0 option to enable best effort communication.
The publisher will now just attempt to deliver the network packets, and don’t expect acknowledgement from the consumer.
We see now that some of the frame on the
showimage side were dropped, the frame numbers in the shell running
showimage won’t be consecutive anymore:
When you’re done, remember to delete the queueing discipline:
sudo tc qdisc delete dev lo root netem loss 5%