Use your phone as a sensor in ROS2

While many projects exist to control your robot from your phone, this project is the other way around; your phone is the robot’ sensors ! It will send the camera feed, IMU and GPS so that you may integrate the phone onto a mobile base.

The particularity of this project is that it relies on the mobile browser instead of a specific app. A webpage is served from the ROS2 node, opening the page from a mobile client will ask for permissions in the browser. The data is transmitted between the server and client using websockets for a modern and fast communication, as opposed to making HTTP requests or streaming UDP.

This repository is inspired by a project I did with students as a TA called phone-imu.

Build

source /opt/ros/humble/setup.bash
rosdep install -i --from-path src --rosdistro humble -y --ignore-src
colcon build --packages-up-to phone_sensors_bridge
# Or build for development, but remember to clean and rebuild every time the JS files are changed
# colcon build --symlink-install  --packages-up-to phone_sensors_bridge_examples --event-handlers console_direct+

phone_sensors_bridge usage

Quickstart

Run the following in a different terminal than the one used for building, otherwise the server might start from inside the build folder which will not contain the template and static folders for the webpage.

source install/setup.bash

# Ubuntu IP
EXTRA_IP=$(ip route get 8.8.8.8 | grep -oP 'src \K[^ ]+')
echo "My IP is $EXTRA_IP"
# Generate SSL certificates for local webserver 
ros2 run phone_sensors_bridge generate_dev_certificates.sh $EXTRA_IP

# Start server
ros2 run phone_sensors_bridge server --ros-args -p video_width:=1280 -p video_height:=720

Open the webpage from your mobile device. The URL contains the server host IP where the node is running. It depends on your network, but most likely is https://<EXTRA_IP>:2000. If you have multiple network interfaces, favour the fastest such as ethernet over wifi.

The page will prompt for permissions, then display the chosen camera

Parameters

A negative value for the time reference, IMU, GNSS frequencies or video FPS will disable sending the corresponding data from the client device. This allows conserving bandwidth and processing power when certain sensors are not needed.

To find out the available camera_device_label, open the video test page

Coordinates

The device rotation rate and acceleration are defined in the device coordinate frame:

The device orientation is defined in ENU. The angles alpha, beta, gamma we get from the web API are the respective rotations around Z, X, Y. We convert these angles to quaternion by using beta as roll, gamma as pitch, alpha as yaw. The behaviour of a device with its Y-axis pointing up is unstable since the API is only defined between -90° to +90°. See the web API documentation for more details.

Browser compatibility

The current server is tested with Firefox and Chrome.

GeoLocation:

• Issue in Firefox: getting the position silently fails

Video:

• Browsers do not use the same device labels

File truncated at 100 lines see the full file