FOROS : Failover ROS framework

Purpose

FOROS is a ROS2 framework that provides the ability to construct a active-standby cluster based on the RAFT consensus algorithm.

Key Features

| Feature | Description | | —————– | ———————————————————————— | | Node Redundancy | Maintain a cluster of nodes runing in parallel to acheive a common goal. | | State Replication | Replicate a state of nodes for implementing a fault-tolerant service. |

Goals

This framework can tolerate fail-stop failures equal to the cluster size minus the quorum. | Cluster size (N) | Quorum (Q = N / 2 + 1) | Number of fault tolerant nodes (N - Q) | | :————–: | :——————–: | :————————————: | | 1 | 1 | 0 | | 2 | 2 | 0 | | 3 | 2 | 1 | | 4 | 3 | 1 | | 5 | 3 | 2 |

Prerequisites (Ubuntu)

Install ROS2

Supported distributions | OS | Distribution | Foros branch | | ———— | ——————————————— | ———— | | Ubuntu 20.04 | galactic | galactic | | Ubuntu 22.04 | humble | humble |

Install Foros

Option 1. Install Official Packages

for galactic

sudo apt install ros-galactic-foros ros-galactic-foros-examples ros-galactic-foros-inspector

for humble

sudo apt install ros-humble-foros ros-humble-foros-examples ros-humble-foros-inspector

Option 2. Build From Source

Install dependencies

sudo apt install libleveldb-dev libncurses-dev

Build

If you want to install to existing ROS2 workspace, please clone this source in the workspace in advance.

. /opt/ros/galactic/setup.bash
colcon build

Getting Started

In this section, we will create a cluster of 3 nodes that publishes a node ID every second. Lets’ assume that, | Cluster Name | Cluster Size | Node IDs in the Cluster | Topic Name | | :————-: | :———-: | :———————: | :————-: | | “hello_cluster” | 3 | 0, 1, 2 | “hello_cluster” |

1) Setup Environment

. /opt/ros/galactic/setup.bash
# If foros is installed in the custom workspace,
. {custom workspace}/install/setup.bash

2) Create ROS2 package

Let’s create the hello_cluster package that depends on foros, rclcpp, and std_msgs.

ros2 pkg create hello_cluster --description "Hello Cluster" --build-type ament_cmake --dependencies foros rclcpp std_msgs

3) Create “src/hello_cluster.cpp”

Most of the code is the same as the general ros2 code, except that the akit::failover::foros::ClusterNode class is used instead of rclcpp::Node when creating a node. Unlike rclcpp::Node that receives a node name as an argument, akit::failover::foros::ClusterNode receives a cluster name, node ID, and IDs of all nodes in the cluster as arguments. In fact, the node name of a cluster node is internally created by combining the cluster name and node ID.

For instance, if cluster name is “hello_cluster” and node id is 0, node name is created as “hello_cluster0”

// hello-cluster.cpp
#include <chrono>
#include <rclcpp/rclcpp.hpp>
#include <std_msgs/msg/string.hpp>
#include <string>
#include <vector>

#include "akit/failover/foros/cluster_node.hpp"

using namespace std::chrono_literals;

int main(int argc, char **argv) {
  const std::string kClusterName = "hello_cluster";
  const std::string kTopicName = "hello_cluster";
  const std::vector<uint32_t> kClusterNodeIds = {0, 1, 2};

  rclcpp::Logger logger = rclcpp::get_logger(argv[0]);
  logger.set_level(rclcpp::Logger::Level::Info);

  if (argc < 2) {
    RCLCPP_ERROR(logger, "Usage : %s {node ID} {size of cluster}", argv[0]);
    return -1;
  }

  uint32_t id = std::stoul(argv[1]);
  if (find(kClusterNodeIds.begin(), kClusterNodeIds.end(), id) ==
      kClusterNodeIds.end()) {
    RCLCPP_ERROR(logger, "Node ID must be among 0, 1, 2");
    return -1;
  }

  rclcpp::init(argc, argv);

  // Create cluster node not rclcpp::Node
  auto node = akit::failover::foros::ClusterNode::make_shared(kClusterName, id,
                                                              kClusterNodeIds);

  auto publisher = node->create_publisher<std_msgs::msg::String>(kTopicName, 1);
  auto timer_ =
      rclcpp::create_timer(node, rclcpp::Clock::make_shared(), 1s, [&]() {
        auto msg = std_msgs::msg::String();
        msg.data = std::to_string(id);
        publisher->publish(msg);
      });
  rclcpp::spin(node->get_node_base_interface());
  rclcpp::shutdown();

  return 0;
}

4) Update “CMakeLists.txt”

Add below code to CMakeLists.txt

# Add "hello_cluster" exectuable target
add_executable(${PROJECT_NAME} src/hello_cluster.cpp)
# Add dependencies
ament_target_dependencies(${PROJECT_NAME}
  foros
  rclcpp
  std_msgs
)
# Install
install(
  TARGETS ${PROJECT_NAME} EXPORT ${PROJECT_NAME}
  DESTINATION lib/${PROJECT_NAME}
)

5) Build

# in the workspace,
colcon build --symlink-install

6) Test

Prior to run the nodes, let’s subscribe to “hello_cluster” topic using ros2 command.

# Subscriber
ros2 topic echo "hello_cluster" std_msgs/String

If you run node 0, node 1, and node 2,

# Node 0
ros2 run hello_cluster hello_cluster 0

# Node 1
ros2 run hello_cluster hello_cluster 1

# Node 2
ros2 run hello_cluster hello_cluster 2

One node (0) became the leader, and it starts to publish mesage to topic.

# Subscriber
## Assume that new leader is node 0
---
data: '0'
---
data: '0'
---
...

When leader node (0) is terminated, one node (1) becomes the leader and starts publishing messages.

# Subscriber
## Assume that new leader is node 1
---
data: '1'
---
data: '1'
---
...

When leader node (1) is terminated, the number of terminated nodes has exceeded fault tolerance and the remaining node (2) has failed to become leader.

# Subscriber
## No output since there is no leader

Restarting node (1) causes one node (2) to become the leader and start publishing messages.

# Subscriber
## Assume that new leader is node 2
---
data: '2'
---
data: '2'
---
...

You can also check the cluster status with the inspector tool as shown below.