Table of Contents
- Open Robotics provides pre-built ROS2 packages for multiple platforms, but a number of developers still rely on cross-compilation for different reasons such as:
The development machine does not match the target system.
Tuning the build for specific core architecture (e.g. setting -mcpu=cortex-a53 -mfpu=neon-fp-armv8 when building for Raspberry Pi3).
Targeting a different file systems other than the ones supported by the pre-built images released by Open Robotics.
This document provides you with details on how to cross-compile the ROS2 software stack as well as provide examples for cross-compiling to systems based on the Arm cores.
Cross-compiling simple software (e.g. no dependencies on external libraries) is relatively simple and only requiring a cross-compiler toolchain to be used instead of the native toolchain.
- There are a number of factors which make this process more complex:
The software being built must support the target architecture. Architecture specific code must be properly isolated and enabled during the build according to the target architecture. Examples include assembly code.
All dependencies (e.g. libraries) must be present, either as pre-built packages or also cross-compiled before the target software using them is cross-compiled.
When building software stacks (as opposed to an standalone software) using build tools (e.g. colcon), it is expected from the build tool a mechanism to allow the developer to enable cross-compilation on the underlying build system used by each of software in the stack.
The ROS2 cross-compile tool is under shared ownership of Open Robotics and ROS Tooling Working Group.
It is a Python script that compiles ROS2 source files for supported target architectures using an emulator in a docker container. Detailed design of the tool can be found on ROS2 design. Instructions to use the tool are in the cross_compile package.
If you are using an older version, please follow the legacy tool instructions.
- Although ROS2 is a rich software stack with a number of dependencies, it primarily uses two different types of packages:
Python based software, which requires no cross-compilation.
CMake based software, which provides a mechanism to do cross-compilation.
Furthermore, the ROS2 software stack is built with Colcon which provides a mechanism to forward parameters to the CMake instance used for the individual build of each package/library that is part of the ROS2 distribution.
When building ROS2 natively, the developer is required to download all the dependencies (e.g. Python and other libraries) before compiling the packages that are part of the ROS2 distribution. When cross-compiling, the same approach is required. The developer must first have the target system’s filesystem with all dependencies already installed.
A CMake toolchain-file is a file which defines variables to configure CMake for cross-compilation. The basic entries are:
CMAKE_SYSTEM_NAME: the target platform, e.g.
CMAKE_SYSTEM_PROCESSOR: the target architecture, e.g.
CMAKE_SYSROOT: the path to the target file-system
CMAKE_C_COMPILER: the C cross-compiler, e.g.
CMAKE_CXX_COMPILER: the C++ cross-compiler, e.g.
CMAKE_FIND_ROOT_PATH: an alternative path used by the
find_*command to find the file-system
When cross-compiling ROS2, the following options are required to be set:
CMAKE_FIND_ROOT_PATH: the alternative path used by the
find_*command, use it to specify the path to ROS2
CMAKE_FIND_ROOT_PATH_MODE_*: the search strategy for program,package,library, and include, usually:
NEVER(look on the host-fs),
ONLY(look on sysroot), and
BOTH(look on both sysroot and host-fs)
PYTHON_SOABI: the index name of the python libraries generated by ROS2, e.g.
THREADS_PTHREAD_ARG "0" CACHE STRING "Result from TRY_RUN" FORCE: Force the result of the
TRY_RUNcmd to 0 (success) because binaries can not run on the host system.
The toolchain-file is provided to CMake with the
-DCMAKE_TOOLCHAIN_FILE=path/to/file parameter. This will also set the
CMAKE_CROSSCOMPILING variable to
true which can be used by the software being built.
CMAKE_SYSROOT is particularly important for ROS2 as the packages need many dependencies (e.g. python, openssl, opencv, poco, eigen3, …).
CMAKE_SYSROOT to a target file-system with all the dependencies installed on it will allow CMake to find them during the cross-compilation.
You can find more information on the CMake documentation page.
When downloading the ROS2 source code, a generic toolchain-file is available in the repository ros-tooling/cross_compile/cmake-toolchains which can be downloaded separately. Further examples on using it can be found on the Cross-compiling examples for Arm section.
As mentioned previously, ROS2 requires different libraries which needs to be provided to cross-compile.
- There are a number of ways to obtain the file-system:
downloading a pre-built image
installing the dependencies on the target and exporting the file-system (e.g. with sshfs)
using qemu + docker (or chroot) to generate the file-system on the host machine.
You can find information on how to use Docker + qemu on the next Cross-compiling examples for Arm section.
The build process is similar to native compilation. The only difference is an extra argument to
Colcon to specify the
colcon build --merge-install \ --cmake-force-configure \ --cmake-args \ -DCMAKE_TOOLCHAIN_FILE="<path_to_toolchain/toolchainfile.cmake>"
toolchain-file provide to CMake the information of the
cross-compiler and the
Colcon will call CMake with the given toolchain-file on every package of ROS2.
After downloading the ROS2 source code, you can add cross-compilation assets to the workspace via
git clone https://github.com/ros-tooling/cross_compile.git -b 0.0.1 src/ros2/cross_compile. These are working examples on how to cross-compile for Arm cores.
- The following targets are supported:
Ubuntu-arm64: To be used with any ARMv8-A based system.
Ubuntu-armhf: To be used with any modern ARMv7-A based system.
- These are the main steps:
Installing development tools
Downloading ROS2 source code
Downloading the ROS2 cross-compilation assets
Preparing the sysroot
Cross-compiling the ROS2 software stack
The next sections explains in detail each of these steps. For a quick-setup, have a look at the Automated Cross-compilation.
These steps were tested on an Ubuntu 18.04 (Bionic)
1. Install development tools¶
This step is similar to when building natively. The difference is that some of the libraries and tools are not required because they will be in the sysroot instead. The following packages are required
sudo apt update && sudo apt install -y \ cmake \ git \ wget \ python3-pip \ qemu-user-static \ g++-aarch64-linux-gnu \ g++-arm-linux-gnueabihf \ pkg-config-aarch64-linux-gnu python3 -m pip install -U \ vcstool \ colcon-common-extensions
You can install vcstool and colcon-common-extensions via pip. This means you are not required to add extra apt repositories.
Docker is used to build the target environment. Follow the official documentation for the installation.
2. Download ROS2 source code¶
Then create a workspace and download the ROS2 source code:
mkdir -p ~/cc_ws/ros2_ws/src cd ~/cc_ws/ros2_ws wget https://raw.githubusercontent.com/ros2/ros2/release-latest/ros2.repos vcs-import src < ros2.repos git clone https://github.com/ros-tooling/cross_compile.git -b 0.0.1 src/ros2/cross_compile cd ..
3. Prepare the sysroot¶
Build an arm Ubuntu image with all the ROS2 dependencies using Docker and qemu:
qemu-static binary to the workspace.
It will be used to install the ros2 dependencies on the target file-system with docker.
mkdir qemu-user-static cp /usr/bin/qemu-*-static qemu-user-static
The standard setup process of ROS2 is run inside an arm docker. This is possible thanks to
qemu-static, which will emulate an arm machine. The base image used is an Ubuntu Bionic from Docker Hub.
docker build -t arm_ros2:latest -f ros2_ws/src/ros2/cross_compile/sysroot/Dockerfile_ubuntu_arm . docker run --name arm_sysroot arm_ros2:latest
Export the resulting container to a tarball and extract it:
docker container export -o sysroot_docker.tar arm_sysroot mkdir sysroot_docker tar -C sysroot_docker -xf sysroot_docker.tar lib usr opt etc docker rm arm_sysroot
This container can be used later as virtual target to run the created file-system and run the demo code.
Set the variables used by the generic toolchain-file
export TARGET_ARCH=aarch64 export TARGET_TRIPLE=aarch64-linux-gnu export CC=/usr/bin/$TARGET_TRIPLE-gcc export CXX=/usr/bin/$TARGET_TRIPLE-g++ export CROSS_COMPILE=/usr/bin/$TARGET_TRIPLE- export SYSROOT=~/cc_ws/sysroot_docker export ROS2_INSTALL_PATH=~/cc_ws/ros2_ws/install export PYTHON_SOABI=cpython-36m-$TARGET_TRIPLE
The following packages still cause errors during the cross-compilation (under investigation) and must be disabled for now.
touch \ ros2_ws/src/ros2/rviz/COLCON_IGNORE \ ros2_ws/src/ros-visualization/COLCON_IGNORE
Poco pre-built has a known issue where it is searching for
libpcre on the host system instead of SYSROOT.
As a workaround for the moment, please link both libraries into the the host’s file-system.
mkdir -p /usr/lib/$TARGET_TRIPLE ln -s `pwd`/sysroot_docker/lib/$TARGET_TRIPLE/libz.so.1 /usr/lib/$TARGET_TRIPLE/libz.so ln -s `pwd`/sysroot_docker/lib/$TARGET_TRIPLE/libpcre.so.3 /usr/lib/$TARGET_TRIPLE/libpcre.so
Then, start a build with colcon specifying the toolchain-file:
cd ros2_ws colcon build --merge-install \ --cmake-force-configure \ --cmake-args \ -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \ -DCMAKE_TOOLCHAIN_FILE="$(pwd)/src/ros2/cross_compile/cmake-toolchains/generic_linux.cmake" \ -DSECURITY=ON
Done! The install and build directories will contain the cross-compiled assets.
All the steps above are also included into a Dockerfile and can be used for automation/CI.
First, download the dockerfile and build the image:
wget https://raw.githubusercontent.com/ros-tooling/cross_compile/master/Dockerfile_cc_for_arm docker build -t ros2-crosscompiler:latest - < Dockerfile_cc_for_arm
Now run the image with: (it will take a while !)
docker run -it --name ros2_cc \ -v /var/run/docker.sock:/var/run/docker.sock \ ros2-crosscompiler:latest
..note:: The -v /var/run/docker.sock allow us to use Docker inside Docker.
The result of the build will be inside the
ros2_ws directory, which can be exported with:
docker cp ros2_cc:/root/cc_ws/ros2_ws .
It is possible to cross-compile your packages against a pre-built ROS2. The steps are similar to the previous Cross-compiling examples for Arm section, with the following modifications:
Instead of downloading the ROS2 stack, just populate your workspace with your package (ros2 examples on this case) and the cross-compilation assets:
mkdir -p ~/cc_ws/ros2_ws/src cd ~/cc_ws/ros2_ws/src git clone https://github.com/ros2/examples.git git clone https://github.com/ros-tooling/cross_compile.git -b 0.0.1 cd ..
Generate and export the file-system as described in 3. Prepare the sysroot, but with the provided
_prebuilt Dockerfile will use the binary packages to install ROS2 instead of building from source.
Modify the environment variable
ROS2_INSTALL_PATH to point to the installation directory:
setup.bash script on the target file-system:
Then, start a build with
Colcon specifying the
colcon build \ --merge-install \ --cmake-force-configure \ --cmake-args \ -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \ -DCMAKE_TOOLCHAIN_FILE="$(pwd)/src/cross_compile/cmake-toolchains/generic_linux.cmake"
Copy the file-system on your target or use the previously built docker image:
docker run -it --rm -v `pwd`/ros2_ws:/ros2_ws arm_ros2:latest
Source the environment:
Run some of the C++ or python examples:
ros2 run demo_nodes_cpp listener & ros2 run demo_nodes_py talker