Repository Summary
| Checkout URI | https://github.com/intel/ros2_object_analytics.git |
| VCS Type | git |
| VCS Version | master |
| Last Updated | 2019-09-16 |
| Dev Status | MAINTAINED |
| CI status | No Continuous Integration |
| Released | RELEASED |
| Tags | No category tags. |
| Contributing |
Help Wanted (0)
Good First Issues (0) Pull Requests to Review (0) |
Packages
| Name | Version |
|---|---|
| object_analytics_msgs | 0.5.4 |
| object_analytics_node | 0.5.4 |
| object_analytics_rviz | 0.5.4 |
README
ros2_object_analytics
Object Analytics (OA) is ROS2 module for Realtime object tracking and 3D localization. These packages aim to provide real-time object analyses over RGB-D camera inputs, enabling ROS developer to easily create amazing robotics advanced features, like intelligent collision avoidance, people follow and semantic SLAM. It consumes sensor_msgs::PointClould2 data delivered by RGB-D camera, subscribes topic on object detection by ros2_intel_movidius_ncs, publishes topics on object tracking in 2D RGB image and object localization in 3D camera coordination system.
OA keeps integrating with various "state-of-the-art" algorithms. * Object detection offload to VPU, Intel Movidius NCS, with MobileNet SSD model and Caffe framework(TODO).
System Requirements
We support Ubuntu Linux Bionic Beaver 18.04 on 64-bit. We not support Mac OS X and Windows.
Hardware Requirements
- Intel NUC (CPU: Intel i7-6700HQ @2.60GHz, Mem:16G)
- Intel Movidius Neural Compute Stick
- Intel RealSense D435i/D435/D415
Dependencies
Install ROS2 desktop packages ros-dashing-desktop
sudo apt-get install ros-dashing-desktop #Follow instructions above in the ROS docs if this doesn't work
The ros-dashing-desktop will include below packages. * ament_cmake * std_msgs * sensor_msgs * geometry_msgs * rclcpp * rosidl_default_generators * rosidl_interface_packages * launch * ros2run * class_loader * pcl_conversions
Install ROS2 dependences
sudo apt-get install ros-dashing-cv-bridge ros-dashing-object-msgs ros-dashing-image-transport ros-dashing-librealsense2 ros-dashing-realsense-camera-msgs ros-dashing-realsense-ros2-camera
- cv_bridge
- object_msgs
- ros2_message_filters
- ros2_intel_realsense (The only supported RGB-D camera by now is Intel RealSense)
Install ros2_intel_movidius
ros2_intel_movidius has not integrated in ROS2 release, so there is no debian package available for Movidius NCS installation, need to build from source, and it should be installed before OpenCV3, more details please refer to https://github.com/intel/ros2_intel_movidius_ncs).
# Build ncsdk
mkdir ~/code
cd ~/code
git clone https://github.com/movidius/ncsdk
git clone https://github.com/movidius/ncappzoo
cd ~/code/ncsdk
make install
ln -sf ~/code/ncappzoo /opt/movidius/ncappzoo
# Build ros2_intel_movidius_ncs
mkdir ~/ros2_ws/src -p
cd ~/ros2_ws/src
git clone https://github.com/intel/ros2_intel_movidius_ncs.git
cd ~/ros2_ws
source /opt/ros/dashing/setup.bash
colcon build --symlink-install (Install python3-colcon-common-extensions by apt-get if colcon command not exist)
# Build CNN model (Please plugin NCS device on the host while compiling)
cd /opt/movidius/ncappzoo/caffe/SSD_MobileNet
make
# Copy object label file to NCSDK installation location.
cp ~/ros2_ws/src/ros2_intel_movidius_ncs/data/labels/* /opt/movidius/ncappzoo/data/ilsvrc12/
Build OpenCV3
OpenCV3 & Opencv-Contrib 3.3 (OA depends on tracking feature from OpenCV Contrib 3.3. OpenCV 3.3 is not integrated in ROS2 Dashing release, need to build and install Opencv3 with Opencv-Contrib from source to apply tracking feature)
# Build and Install OpenCV3 with Opencv-Contrib
mkdir ${HOME}/opencv
cd ${HOME}/opencv
git clone https://github.com/opencv/opencv.git -b 3.3.0
git clone https://github.com/opencv/opencv_contrib.git -b 3.3.0
mkdir opencv/build -p
cd opencv/build
cmake -DOPENCV_EXTRA_MODULES_PATH=${HOME}/opencv/opencv_contrib/modules \
-DCMAKE_INSTALL_PREFIX=/usr/local -DBUILD_opencv_cnn_3dobj=OFF ..
make -j8
sudo make install
sudo ldconfig
Install OpenCV3 dependences
sudo apt-get install liblz4-dev
Install OA
Install OA debian packages
sudo apt-get install ros-dashing-object-analytics-msgs ros-dashing-object-analytics-node ros-dashing-object-analytics-rviz
The object analytics packages installation have been completed. You could jump to Run for executing, you could also install OA from source for more features. Notes: debian installed package does not support 2d tracking feature as the dependent opencv3.3 has no debian available. For full feature, please build opencv3.3 and install object analytics from source.
Install OA from source
# Get code
mkdir ~/ros2_ws/src -p
cd ~/ros2_ws/src
git clone https://github.com/intel/ros2_object_analytics.git -b devel (devel branch is the latest code with 2D tracking features, while master branch is stable for ros2 Dashing release)
# Build
cd ~/ros2_ws
source /opt/ros/dashing/setup.bash
colcon build --symlink-install
Run
Configure NCS default.yaml
source /opt/ros/dashing/setup.bash
source ~/ros2_ws/install/local_setup.bash
echo -e "param_file: mobilenetssd.yaml\ninput_topic: /object_analytics/rgb" > `ros2 pkg prefix movidius_ncs_launch`/share/movidius_ncs_launch/config/default.yaml
Start OA Demo
source /opt/ros/dashing/setup.bash
source ~/ros2_ws/install/local_setup.bash
ros2 launch object_analytics_node object_analytics.launch.py
Subscribed topics
/object_analytics/detected_objects (object_msgs::msg::ObjectsInBoxes)
Published topics
/object_analytics/rgb (sensor_msgs::msg::Image)
/object_analytics/pointcloud (sensor_msgs::msg::PointCloud2)
/object_analytics/localization (object_analytics_msgs::msg::ObjectsInBoxes3D)
/object_analytics/tracking (object_analytics_msgs::msg::TrackedObjects)
Customize launch
By default, object analytics will launch both tracking and localization features, but either tracking or localization or both can be dropped. Detailed please refer comments embedded in launch file.
Tools
To ensure the algorithms in OA components to archive best performance in ROS2, we have below tools used to examine design/development performance/accuracy/precision..., more tools are in developing progress and will publish later.
1. tracker_regression
The tools is used to feed tracking node with raw images from datasets within fixed time interval(33ms), also simulate detector send ground truth as detections to tracking node for rectification, then receive tracking results for precision and recall statistics. It support multiple algorithms(dynamic configure to tracking node when start).
* Tools usages
# ros2 run object_analytics_node tracker_regression --options
options: [-a algorithm] [-p dataset_path] [-t dataset_type] [-n dataset_name] [-h];
-h : Print this help function.
-a algorithm_name : Specify the tracking algorithm in the tracker.
supported algorithms: KCF,TLD,BOOSTING,MEDIAN_FLOW,MIL,GOTURN.
-p dataset_path : Specify the tracking datasets location.
-t dataset_type : Specify the dataset type: video,image.
-n dataset_name : Specify the dataset name
* Example:
Video dataset with tracking algorithm("MEDIAN_FLOW"):
# ros2 run object_analytics_node tracker_regression -p /your/video/datasets/root/path -t video -n dudek -a MEDIAN_FLOW
Image dataset with default algorithm("MEDIAN_FLOW"):
# ros2 run object_analytics_node tracker_regression -p /your/image/datasets/root/path -t image -n Biker -a MEDIAN_FLOW
* Dataset:
Support both video and image dataset, but you may need to translate into below formats.
Video dataset: (Refer to opencv_extra tracking dataset)
track_vid/ (/your/video/datasets/root/path)
├── david
│ ├── data
│ │ └── david.webm
│ ├── david.yml
│ ├── gt.txt
│ └── initOmit
│ └── david.txt
├── dudek
│ ├── data
│ │ └── dudek.webm
│ ├── dudek.yml
│ ├── gt.txt
│ └── initOmit
│ └── dudek.txt
├── faceocc2
│ ├── data
│ │ └── faceocc2.webm
│ ├── faceocc2.yml
│ ├── gt.txt
│ └── initOmit
│ └── faceocc2.txt
├── list.txt (Note: this is manually added, list the dataset names which will be used)
└── README.md
Image dataset: (Refer to database from Computer Vision Lab@HYU)
track_img/ (/your/video/datasets/root/path)
├── Biker
├── Bird1
├── Bird2
├── list.txt (Note: this is manually added, list the dataset names which will be used)
├── Man
├── Matrix
└── Woman
Any security issue should be reported using process at https://01.org/security
CONTRIBUTING
Repository Summary
| Checkout URI | https://github.com/intel/ros2_object_analytics.git |
| VCS Type | git |
| VCS Version | master |
| Last Updated | 2019-09-16 |
| Dev Status | MAINTAINED |
| CI status | No Continuous Integration |
| Released | RELEASED |
| Tags | No category tags. |
| Contributing |
Help Wanted (0)
Good First Issues (0) Pull Requests to Review (0) |
Packages
| Name | Version |
|---|---|
| object_analytics_msgs | 0.5.4 |
| object_analytics_node | 0.5.4 |
| object_analytics_rviz | 0.5.4 |
README
ros2_object_analytics
Object Analytics (OA) is ROS2 module for Realtime object tracking and 3D localization. These packages aim to provide real-time object analyses over RGB-D camera inputs, enabling ROS developer to easily create amazing robotics advanced features, like intelligent collision avoidance, people follow and semantic SLAM. It consumes sensor_msgs::PointClould2 data delivered by RGB-D camera, subscribes topic on object detection by ros2_intel_movidius_ncs, publishes topics on object tracking in 2D RGB image and object localization in 3D camera coordination system.
OA keeps integrating with various "state-of-the-art" algorithms. * Object detection offload to VPU, Intel Movidius NCS, with MobileNet SSD model and Caffe framework(TODO).
System Requirements
We support Ubuntu Linux Bionic Beaver 18.04 on 64-bit. We not support Mac OS X and Windows.
Hardware Requirements
- Intel NUC (CPU: Intel i7-6700HQ @2.60GHz, Mem:16G)
- Intel Movidius Neural Compute Stick
- Intel RealSense D435i/D435/D415
Dependencies
Install ROS2 desktop packages ros-dashing-desktop
sudo apt-get install ros-dashing-desktop #Follow instructions above in the ROS docs if this doesn't work
The ros-dashing-desktop will include below packages. * ament_cmake * std_msgs * sensor_msgs * geometry_msgs * rclcpp * rosidl_default_generators * rosidl_interface_packages * launch * ros2run * class_loader * pcl_conversions
Install ROS2 dependences
sudo apt-get install ros-dashing-cv-bridge ros-dashing-object-msgs ros-dashing-image-transport ros-dashing-librealsense2 ros-dashing-realsense-camera-msgs ros-dashing-realsense-ros2-camera
- cv_bridge
- object_msgs
- ros2_message_filters
- ros2_intel_realsense (The only supported RGB-D camera by now is Intel RealSense)
Install ros2_intel_movidius
ros2_intel_movidius has not integrated in ROS2 release, so there is no debian package available for Movidius NCS installation, need to build from source, and it should be installed before OpenCV3, more details please refer to https://github.com/intel/ros2_intel_movidius_ncs).
# Build ncsdk
mkdir ~/code
cd ~/code
git clone https://github.com/movidius/ncsdk
git clone https://github.com/movidius/ncappzoo
cd ~/code/ncsdk
make install
ln -sf ~/code/ncappzoo /opt/movidius/ncappzoo
# Build ros2_intel_movidius_ncs
mkdir ~/ros2_ws/src -p
cd ~/ros2_ws/src
git clone https://github.com/intel/ros2_intel_movidius_ncs.git
cd ~/ros2_ws
source /opt/ros/dashing/setup.bash
colcon build --symlink-install (Install python3-colcon-common-extensions by apt-get if colcon command not exist)
# Build CNN model (Please plugin NCS device on the host while compiling)
cd /opt/movidius/ncappzoo/caffe/SSD_MobileNet
make
# Copy object label file to NCSDK installation location.
cp ~/ros2_ws/src/ros2_intel_movidius_ncs/data/labels/* /opt/movidius/ncappzoo/data/ilsvrc12/
Build OpenCV3
OpenCV3 & Opencv-Contrib 3.3 (OA depends on tracking feature from OpenCV Contrib 3.3. OpenCV 3.3 is not integrated in ROS2 Dashing release, need to build and install Opencv3 with Opencv-Contrib from source to apply tracking feature)
# Build and Install OpenCV3 with Opencv-Contrib
mkdir ${HOME}/opencv
cd ${HOME}/opencv
git clone https://github.com/opencv/opencv.git -b 3.3.0
git clone https://github.com/opencv/opencv_contrib.git -b 3.3.0
mkdir opencv/build -p
cd opencv/build
cmake -DOPENCV_EXTRA_MODULES_PATH=${HOME}/opencv/opencv_contrib/modules \
-DCMAKE_INSTALL_PREFIX=/usr/local -DBUILD_opencv_cnn_3dobj=OFF ..
make -j8
sudo make install
sudo ldconfig
Install OpenCV3 dependences
sudo apt-get install liblz4-dev
Install OA
Install OA debian packages
sudo apt-get install ros-dashing-object-analytics-msgs ros-dashing-object-analytics-node ros-dashing-object-analytics-rviz
The object analytics packages installation have been completed. You could jump to Run for executing, you could also install OA from source for more features. Notes: debian installed package does not support 2d tracking feature as the dependent opencv3.3 has no debian available. For full feature, please build opencv3.3 and install object analytics from source.
Install OA from source
# Get code
mkdir ~/ros2_ws/src -p
cd ~/ros2_ws/src
git clone https://github.com/intel/ros2_object_analytics.git -b devel (devel branch is the latest code with 2D tracking features, while master branch is stable for ros2 Dashing release)
# Build
cd ~/ros2_ws
source /opt/ros/dashing/setup.bash
colcon build --symlink-install
Run
Configure NCS default.yaml
source /opt/ros/dashing/setup.bash
source ~/ros2_ws/install/local_setup.bash
echo -e "param_file: mobilenetssd.yaml\ninput_topic: /object_analytics/rgb" > `ros2 pkg prefix movidius_ncs_launch`/share/movidius_ncs_launch/config/default.yaml
Start OA Demo
source /opt/ros/dashing/setup.bash
source ~/ros2_ws/install/local_setup.bash
ros2 launch object_analytics_node object_analytics.launch.py
Subscribed topics
/object_analytics/detected_objects (object_msgs::msg::ObjectsInBoxes)
Published topics
/object_analytics/rgb (sensor_msgs::msg::Image)
/object_analytics/pointcloud (sensor_msgs::msg::PointCloud2)
/object_analytics/localization (object_analytics_msgs::msg::ObjectsInBoxes3D)
/object_analytics/tracking (object_analytics_msgs::msg::TrackedObjects)
Customize launch
By default, object analytics will launch both tracking and localization features, but either tracking or localization or both can be dropped. Detailed please refer comments embedded in launch file.
Tools
To ensure the algorithms in OA components to archive best performance in ROS2, we have below tools used to examine design/development performance/accuracy/precision..., more tools are in developing progress and will publish later.
1. tracker_regression
The tools is used to feed tracking node with raw images from datasets within fixed time interval(33ms), also simulate detector send ground truth as detections to tracking node for rectification, then receive tracking results for precision and recall statistics. It support multiple algorithms(dynamic configure to tracking node when start).
* Tools usages
# ros2 run object_analytics_node tracker_regression --options
options: [-a algorithm] [-p dataset_path] [-t dataset_type] [-n dataset_name] [-h];
-h : Print this help function.
-a algorithm_name : Specify the tracking algorithm in the tracker.
supported algorithms: KCF,TLD,BOOSTING,MEDIAN_FLOW,MIL,GOTURN.
-p dataset_path : Specify the tracking datasets location.
-t dataset_type : Specify the dataset type: video,image.
-n dataset_name : Specify the dataset name
* Example:
Video dataset with tracking algorithm("MEDIAN_FLOW"):
# ros2 run object_analytics_node tracker_regression -p /your/video/datasets/root/path -t video -n dudek -a MEDIAN_FLOW
Image dataset with default algorithm("MEDIAN_FLOW"):
# ros2 run object_analytics_node tracker_regression -p /your/image/datasets/root/path -t image -n Biker -a MEDIAN_FLOW
* Dataset:
Support both video and image dataset, but you may need to translate into below formats.
Video dataset: (Refer to opencv_extra tracking dataset)
track_vid/ (/your/video/datasets/root/path)
├── david
│ ├── data
│ │ └── david.webm
│ ├── david.yml
│ ├── gt.txt
│ └── initOmit
│ └── david.txt
├── dudek
│ ├── data
│ │ └── dudek.webm
│ ├── dudek.yml
│ ├── gt.txt
│ └── initOmit
│ └── dudek.txt
├── faceocc2
│ ├── data
│ │ └── faceocc2.webm
│ ├── faceocc2.yml
│ ├── gt.txt
│ └── initOmit
│ └── faceocc2.txt
├── list.txt (Note: this is manually added, list the dataset names which will be used)
└── README.md
Image dataset: (Refer to database from Computer Vision Lab@HYU)
track_img/ (/your/video/datasets/root/path)
├── Biker
├── Bird1
├── Bird2
├── list.txt (Note: this is manually added, list the dataset names which will be used)
├── Man
├── Matrix
└── Woman