Edit on Github

Cross-Compilation

Overview

Open Robotics provides pre-built ROS2 packages for multiple platforms, but a number of developers still rely on cross-compilation for different reasons such as:
  • The development machine does not match the target system.

  • Tuning the build for specific core architecture (e.g. setting -mcpu=cortex-a53 -mfpu=neon-fp-armv8 when building for Raspberry Pi3).

  • Targeting a different file systems other than the ones supported by the pre-built images released by Open Robotics.

This document provides you with details on how to cross-compile the ROS2 software stack as well as provide examples for cross-compiling to systems based on the Arm cores.

Note

There are a few ROS2 packages that fail cross-compilation and have to be disabled during the build. See 4. Build section.

How does it work ?

Cross-compiling simple software (e.g. no dependencies on external libraries) is relatively simple and only requiring a cross-compiler toolchain to be used instead of the native toolchain.

There are a number of factors which make this process more complex:
  • The software being built must support the target architecture. Architecture specific code must be properly isolated and enabled during the build according to the target architecture. Examples include assembly code.

  • All dependencies (e.g. libraries) must be present, either as pre-built packages or also cross-compiled before the target software using them is cross-compiled.

  • When building software stacks (as opposed to an standalone software) using build tools (e.g. colcon), it is expected from the build tool a mechanism to allow the developer to enable cross-compilation on the underlying build system used by each of software in the stack.

Cross-compiling ROS2

Although ROS2 is a rich software stack with a number of dependencies, it primarily uses two different types of packages:
  • Python based software, which requires no cross-compilation.

  • CMake based software, which provides a mechanism to do cross-compilation.

Furthermore, the ROS2 software stack is built with Colcon which provides a mechanism to forward parameters to the CMake instance used for the individual build of each package/library that is part of the ROS2 distribution.

When building ROS2 natively, the developer is required to download all the dependencies (e.g. Python and other libraries) before compiling the packages that are part of the ROS2 distribution. When cross-compiling, the same approach is required. The developer must first have the target system’s filesystem with all dependencies already installed.

The next sections of this document explain in detail the use of cmake-toolchains and the CMAKE_SYSROOT feature to cross-compile ROS2.

CMake toolchain-file

A CMake toolchain-file is a file which defines variables to configure CMake for cross-compilation. The basic entries are:

  • CMAKE_SYSTEM_NAME: the target platform, e.g. linux

  • CMAKE_SYSTEM_PROCESSOR: the target architecture, e.g. aarch64 or arm

  • CMAKE_SYSROOT: the path to the target file-system

  • CMAKE_C_COMPILER: the C cross-compiler, e.g. aarch64-linux-gnu-gcc

  • CMAKE_CXX_COMPILER: the C++ cross-compiler, e.g. aarch64-linux-gnu-g++

  • CMAKE_FIND_ROOT_PATH: an alternative path used by the find_* command to find the file-system

When cross-compiling ROS2, the following options are required to be set:

  • CMAKE_FIND_ROOT_PATH: the alternative path used by the find_* command, use it to specify the path to ROS2 /install folder

  • CMAKE_FIND_ROOT_PATH_MODE_*: the search strategy for program,package,library, and include, usually: NEVER (look on the host-fs), ONLY (look on sysroot), ONLY (look on sysroot) and ONLY (look on sysroot)

  • PYTHON_SOABI: the index name of the python libraries generated by ROS2, e.g. cpython-36m-aarch64-linux-gnu

  • THREADS_PTHREAD_ARG "0" CACHE STRING "Result from TRY_RUN" FORCE: Force the result of the TRY_RUN cmd to 0 (success) because binaries can not run on the host system.

The toolchain-file is provided to CMake with the -DCMAKE_TOOLCHAIN_FILE=path/to/file parameter. This will also set the CMAKE_CROSSCOMPILING variable to true which can be used by the software being built.

The CMAKE_SYSROOT is particularly important for ROS2 as the packages need many dependencies (e.g. python, openssl, opencv, poco, eigen3, …). Setting CMAKE_SYSROOT to a target file-system with all the dependencies installed on it will allow CMake to find them during the cross-compilation.

Note

You can find more information on the CMake documentation page.

When dowloading the ROS2 source code, a generic toolchain-file is available in the repository ros2/cross_compile/cmake-toolchains which can be downloaded separately. Further examples on using it can be found on the Cross-compiling examples for Arm section.

Target file-system

As mentioned previously, ROS2 requires different libraries which needs to be provided to cross-compile.

There are a number of ways to obtain the file-system:
  • downloading a pre-built image

  • installing the dependencies on the target and exporting the file-system (e.g. with sshfs)

  • using qemu + docker (or chroot) to generate the file-system on the host machine.

Note

You can find information on how to use Docker + qemu on the next Cross-compiling examples for Arm section.

Build process

The build process is similar to native compilation. The only difference is an extra argument to Colcon to specify the toolchain-file:

colcon build --merge-install \
    --cmake-force-configure \
    --cmake-args \
        -DCMAKE_TOOLCHAIN_FILE="<path_to_toolchain/toolchainfile.cmake>"

The toolchain-file provide to CMake the information of the cross-compiler and the target file-system. Colcon will call CMake with the given toolchain-file on every package of ROS2.

Cross-compiling examples for Arm

After downloading the ROS2 source code, you can add cross-compilation assets to the workspace via git clone https://github.com/ros2/cross_compile.git src/ros2/cross_compile. These are working examples on how to cross-compile for Arm cores.

The following targets are supported:
  • Ubuntu-arm64: To be used with any ARMv8-A based system.

  • Ubuntu-armhf: To be used with any modern ARMv7-A based system.

These are the main steps:
  • Installing development tools

  • Downloading ROS2 source code

  • Downloading the ROS2 cross-compilation assets

  • Preparing the sysroot

  • Cross-compiling the ROS2 software stack

The next sections explains in detail each of these steps. For a quick-setup, have a look at the Automated Cross-compilation.

Note

These steps were tested on an Ubuntu 18.04 (Bionic)

1. Install development tools

This step is similar to when building natively. The difference is that some of the libraries and tools are not required because they will be in the sysroot instead. The following packages are required

sudo apt update && sudo apt install -y \
    cmake \
    git \
    wget \
    python3-pip \
    qemu-user-static \
    g++-aarch64-linux-gnu \
    g++-arm-linux-gnueabihf \
    pkg-config-aarch64-linux-gnu

python3 -m pip install -U \
    vcstool \
    colcon-common-extensions

Note

You can install vcstool and colcon-common-extensions via pip. This means you are not required to add extra apt repositories.

Docker is used to build the target environment. Follow the official documentation for the installation.

2. Download ROS2 source code

Then create a workspace and download the ROS2 source code:

mkdir -p ~/cc_ws/ros2_ws/src
cd ~/cc_ws/ros2_ws
wget https://raw.githubusercontent.com/ros2/ros2/release-latest/ros2.repos
vcs-import src < ros2.repos
git clone https://github.com/ros2/cross_compile.git src/ros2/cross_compile
cd ..

3. Prepare the sysroot

Build an arm Ubuntu image with all the ROS2 dependencies using Docker and qemu: Copy the qemu-static binary to the workspace. It will be used to install the ros2 dependencies on the target file-system with docker.

mkdir qemu-user-static
cp /usr/bin/qemu-*-static qemu-user-static

The standard setup process of ROS2 is run inside an arm docker. This is possible thanks to qemu-static, which will emulate an arm machine. The base image used is an Ubuntu Bionic from Docker Hub.

docker build -t arm_ros2:latest -f ros2_ws/src/ros2/cross_compile/sysroot/Dockerfile_ubuntu_arm .
docker run --name arm_sysroot arm_ros2:latest

Export the resulting container to a tarball and extract it:

docker container export -o sysroot_docker.tar arm_sysroot
mkdir sysroot_docker
tar -C sysroot_docker -xf sysroot_docker.tar lib usr opt etc
docker rm arm_sysroot

This container can be used later as virtual target to run the created file-system and run the demo code.

4. Build

Set the variables used by the generic toolchain-file

export TARGET_ARCH=aarch64
export TARGET_TRIPLE=aarch64-linux-gnu
export CC=/usr/bin/$TARGET_TRIPLE-gcc
export CXX=/usr/bin/$TARGET_TRIPLE-g++
export CROSS_COMPILE=/usr/bin/$TARGET_TRIPLE-
export SYSROOT=~/cc_ws/sysroot_docker
export ROS2_INSTALL_PATH=~/cc_ws/ros2_ws/install
export PYTHON_SOABI=cpython-36m-$TARGET_TRIPLE

The following packages still cause errors during the cross-compilation (under investigation) and must be disabled for now.

touch \
    ros2_ws/src/ros2/rviz/COLCON_IGNORE \
    ros2_ws/src/ros-visualization/COLCON_IGNORE

The Poco pre-built has a known issue where it is searching for libz and libpcre on the host system instead of SYSROOT. As a workaround for the moment, please link both libraries into the the host’s file-system.

mkdir -p /usr/lib/$TARGET_TRIPLE
ln -s `pwd`/sysroot_docker/lib/$TARGET_TRIPLE/libz.so.1 /usr/lib/$TARGET_TRIPLE/libz.so
ln -s `pwd`/sysroot_docker/lib/$TARGET_TRIPLE/libpcre.so.3 /usr/lib/$TARGET_TRIPLE/libpcre.so

Then, start a build with colcon specifying the toolchain-file:

cd ros2_ws

colcon build --merge-install \
    --cmake-force-configure \
    --cmake-args \
        -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
        -DCMAKE_TOOLCHAIN_FILE="$(pwd)/src/ros2/cross_compile/cmake-toolchains/generic_linux.cmake" \
        -DSECURITY=ON

Done! The install and build directories will contain the cross-compiled assets.

Automated Cross-compilation

All the steps above are also included into a Dockerfile and can be used for automation/CI.

First, download the dockerfile and build the image:

wget https://raw.githubusercontent.com/ros2/cross_compile/master/Dockerfile_cc_for_arm
docker build -t ros2-crosscompiler:latest - < Dockerfile_cc_for_arm

Now run the image with: (it will take a while !)

docker run -it --name ros2_cc \
    -v /var/run/docker.sock:/var/run/docker.sock \
    ros2-crosscompiler:latest

..note:: The -v /var/run/docker.sock allow us to use Docker inside Docker.

The result of the build will be inside the ros2_ws directory, which can be exported with:

docker cp ros2_cc:/root/cc_ws/ros2_ws .

Cross-compiling against a pre-built ROS2

It is possible to cross-compile your packages against a pre-built ROS2. The steps are similar to the previous Cross-compiling examples for Arm section, with the following modifications:

Instead of downloading the ROS2 stack, just populate your workspace with your package (ros2 examples on this case) and the cross-compilation assets:

mkdir -p ~/cc_ws/ros2_ws/src
cd ~/cc_ws/ros2_ws/src
git clone https://github.com/ros2/examples.git
git clone https://github.com/ros2/cross_compile.git
cd ..

Generate and export the file-system as described in 3. Prepare the sysroot, but with the provided Dockerfile_ubuntu_arm64_prebuilt. These _prebuilt Dockerfile will use the binary packages to install ROS2 instead of building from source.

Modify the environment variable ROS2_INSTALL_PATH to point to the installation directory:

export ROS2_INSTALL_PATH=~/cc_ws/sysroot_docker/opt/ros/crystal

Source the setup.bash script on the target file-system:

source $ROS2_INSTALL_PATH/setup.bash

Then, start a build with Colcon specifying the toolchain-file:

colcon build \
    --merge-install \
    --cmake-force-configure \
    --cmake-args \
        -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
        -DCMAKE_TOOLCHAIN_FILE="$(pwd)/src/cross_compile/cmake-toolchains/generic_linux.cmake"

Run on the target

Copy the file-system on your target or use the previously built docker image:

docker run -it --rm -v `pwd`/ros2_ws:/ros2_ws arm_ros2:latest

Source the environment:

source /ros2_ws/install/local_setup.bash

Run some of the C++ or python examples:

ros2 run demo_nodes_cpp listener &
ros2 run demo_nodes_py talker