ros2_object_analytics repository

Repository Summary

Checkout URI https://github.com/intel/ros2_object_analytics.git
VCS Type git
VCS Version master
Last Updated 2018-12-07
Dev Status MAINTAINED
Released RELEASED

Packages

README

ros2_object_analytics

Object Analytics (OA) is ROS2 wrapper for realtime object tracking and 3D localization. These packages aim to provide real-time object analyses over RGB-D camera inputs, enabling ROS developer to easily create amazing robotics advanced features, like intelligent collision avoidance, people follow and semantic SLAM. It consumes sensor_msgs::PointClould2 data delivered by RGB-D camera, subscribs topic on object detection by ros2_intel_movidius_ncs, publishs topics on object tracking in 2D RGB image and object localization in 3D camera coordination system.

OA_Architecture

OA keeps integrating with various "state-of-the-art" algorithms. * Object detection offload to VPU, Intel Movidius NCS, with MobileNet SSD model and Caffe framework(TODO).

System Requirements

We support Ubuntu Linux Bionic Beaver 18.04 on 64-bit. We not support Mac OS X and Windows.

Hardware Requirements

  • Intel NUC (CPU: Intel i7-6700HQ @2.60GHz, Mem:16G)
  • Intel Movidius Neural Compute Stick
  • Intel RealSense D435/D415

Dependencies

Install ROS2 packages ros-bouncy-desktop * ament_cmake * std_msgs * sensor_msgs * geometry_msgs * rclcpp * rosidl_default_generators * rosidl_interface_packages * launch * ros2run * class_loader * pcl_conversions * cv_bridge * object_msgs * ros2_message_filters * ros2_intel_realsense (The only supported RGB-D camera by now is Intel RealSense) * ros2_intel_movidius_ncs (Movidius NCS is the only supported detection backend)

Other non-ROS debian packages * libpcl-dev * python3-numpy * OpenCV3 & opencv-contrib 3.3 (OA depends on tracking feature from OpenCV Contrib 3.3. OpenCV 3.3 is not integrated in ROS2 Bouncy release, need to build and install Opencv3 with contrib from source to apply tracking feature)

  # Build and Install OpenCV3 with opencv-contrib
  mkdir ${HOME}/opencv
  cd ${HOME}/opencv
  git clone https://github.com/opencv/opencv.git -b 3.3.0
  git clone https://github.com/opencv/opencv_contrib.git -b 3.3.0
  mkdir opencv/build -p
  cd opencv/build
  cmake -DOPENCV_EXTRA_MODULES_PATH=${HOME}/opencv/opencv_contrib/modules \ 
        -DCMAKE_INSTALL_PREFIX=/usr/local -DBUILD_opencv_cnn_3dobj=OFF ..
  make -j8
  sudo make install
  sudo ldconfig

Get Code

  mkdir ~/ros2_ws/src -p
  cd ~/ros2_ws/src

  git clone https://github.com/ros-perception/vision_opencv.git -b ros2
  git clone https://github.com/intel/ros2_object_msgs.git
  git clone https://github.com/ros2/message_filters.git
  git clone https://github.com/intel/ros2_intel_realsense.git
  git clone https://github.com/intel/ros2_intel_movidius_ncs.git
  git clone https://github.com/intel/ros2_object_analytics.git

Build

  cd ~/ros2_ws/src
  source /opt/ros/bouncy/local_setup.bash
  cd ${ros2_ws} # "ros2_ws" is the workspace root directory where this project is placed in
  colcon build --symlink-install

Run

Configure NCS default.yaml

  source /opt/ros/bouncy/local_setup.bash
  source ~/ros2_ws/local_setup.bash
  echo -e "param_file: mobilenetssd.yaml\ninput_topic: /object_analytics/rgb" > `ros2 pkg prefix movidius_ncs_launch`/share/movidius_ncs_launch/config/default.yaml

Start OA Demo

  source /opt/ros/bouncy/local_setup.bash
  source ~/ros2_ws/local_setup.bash
  ros2 launch object_analytics_node object_analytics.launch.py

OA_demo_video

Subscribed topics

/movidius_ncs_stream/detected_objects (object_msgs::msg::ObjectsInBoxes)

Published topics

/object_analytics/rgb (sensor_msgs::msg::Image)

/object_analytics/pointcloud (sensor_msgs::msg::PointCloud2)

/object_analytics/localization (object_analytics_msgs::msg::ObjectsInBoxes3D)

/object_analytics/tracking (object_analytics_msgs::msg::TrackedObjects)

/object_analytics/movement (object_analytics_msgs::msg::MovingObjectsInFrame)

Customize launch

By default, object analytics will launch both tracking and localization features, but either tracking or localization or both can be dropped. Detailed please refer comments embedded in launch file.

Tools

To ensure the algorims in OA components to archive best performance in ROS2, we have below tools used to examine design/development performance/accuracy/precision..., more tools are in developing progress and will publish later.

1. tracker_regression

The tools is used to feed tracking node with raw images from datasets within fixed time interval(33ms), also simulate detector send groundtruth as detections to tracking node for rectification, then receive tracking results for precision and recall stastics. It support multiple algorithms(dynamic configure to tracking node when start).

* Tools usages

# ros2 run object_analytics_node tracker_regression --options
       options: [-a algorithm] [-p dataset_path] [-t dataset_type] [-n dataset_name] [-h];
       -h : Print this help function.
       -a algorithm_name : Specify the tracking algorithm in the tracker.
          supported algorithms: KCF,TLD,BOOSTING,MEDIAN_FLOW,MIL,GOTURN.
       -p dataset_path : Specify the tracking datasets location.
       -t dataset_type : Specify the dataset type: video,image.
       -n dataset_name : Specify the dataset name

* Example:

Video dataset with tracking algorithm("MEDIAN_FLOW"):
# ros2 run object_analytics_node tracker_regression -p /your/video/datasets/root/path -t video -n dudek -a MEDIAN_FLOW

Image dataset with default algorithm("MEDIAN_FLOW"):
# ros2 run object_analytics_node tracker_regression -p /your/image/datasets/root/path -t image -n Biker -a MEDIAN_FLOW

* Dataset:

Support both video and image dataset, but you may need to translate into below formats.

Video dataset: (Refer to opencv_extra tracking dataset)

 track_vid/    (/your/video/datasets/root/path)
       ├── david
       │   ├── data
       │   │   └── david.webm
       │   ├── david.yml
       │   ├── gt.txt
       │   └── initOmit
       │       └── david.txt
       ├── dudek
       │   ├── data
       │   │   └── dudek.webm
       │   ├── dudek.yml
       │   ├── gt.txt
       │   └── initOmit
       │       └── dudek.txt
       ├── faceocc2
       │   ├── data
       │   │   └── faceocc2.webm
       │   ├── faceocc2.yml
       │   ├── gt.txt
       │   └── initOmit
       │       └── faceocc2.txt
       ├── list.txt (Note: this is mannually added, list the dataset names which will be used)
       └── README.md

Image dataset: (Refer to database from Computer Vision Lab@HYU)

 track_img/    (/your/video/datasets/root/path)
       ├── Biker
       ├── Bird1
       ├── Bird2
       ├── list.txt (Note: this is mannually added, list the dataset names which will be used)
       ├── Man
       ├── Matrix
       └── Woman
Any security issue should be reported using process at https://01.org/security