nicla_vision_ros package from nicla_vision_ros reponicla_vision_ros |
|
Package Summary
Tags | No category tags. |
Version | 1.0.2 |
License | Apache License 2.0 |
Build type | CATKIN |
Use | RECOMMENDED |
Repository Summary
Checkout URI | https://github.com/ADVRHumanoids/nicla_vision_ros.git |
VCS Type | git |
VCS Version | master |
Last Updated | 2024-10-11 |
Dev Status | MAINTAINED |
CI status | Continuous Integration |
Released | RELEASED |
Tags | No category tags. |
Contributing |
Help Wanted (0)
Good First Issues (0) Pull Requests to Review (0) |
Package Description
Additional Links
Maintainers
- Davide Torielli
- Damiano Gasperini
Authors
- Davide Torielli
- Damiano Gasperini
- Edoardo Del Bianco
- Federico Rollo
Nicla Vision ROS package
:rocket: Check out the ROS2 version: Nicla Vision ROS2 repository :rocket:
Description
This ROS package enables the Arduino Nicla Vision board to be ready-to-use in the ROS world! :boom:
The implemented architecture is described in the above image: the Arduino Nicla Vision board streams the sensors data to a ROS-running machine through TCP/UDP socket. This package will be running on the ROS-running machine, allowing to deserialize the received info, and stream it in the corresponding ROS topics
Here a list of the available sensors with their respective ROS topics:
-
2MP color camera streams on
/nicla/camera/camera_info
/nicla/camera/image_raw
/nicla/camera/image_raw/compressed
-
Time-of-Flight (distance) sensor streams on:
/nicla/tof
-
Microphone streams on:
/nicla/audio
/nicla/audio_info
/nicla/audio_stamped
/nicla/audio_recognized
-
Imu streams on:
/nicla/imu
The user can easily configure this package, by launch parameters, to receive sensors data via either UDP or TCP socket connections, specifying also the socket IP address. Moreover, the user can decide which sensor to be streamed within the ROS environment. In this repository you can find the Python code optimised for receiving the data by the board, and subsequently publishing it through ROS topics.
Table of Contents
Installation
Step-by-step instructions on how to get the ROS package running
Binaries available for noetic
:
sudo apt install ros-$ROS_DISTRO-nicla-vision-ros
For ROS2, check https://github.com/ADVRHumanoids/nicla_vision_ros2.git
Source installation
Usual catkin
build :
$ cd <your_workpace>/src
$ git clone https://github.com/ADVRHumanoids/nicla_vision_ros.git
$ cd <your_workpace>
$ catkin build
$ source <your_workpace>/devel/setup.bash
Arduino Nicla Vision setup
After having completed the setup steps in the Nicla Vision Drivers repository, just turn on your Arduino Nicla Vision. When you power on your Arduino Nicla Vision, it will automatically connect to the network and it will start streaming to your ROS-running machine.
Optional Audio Recognition with VOSK
It is possible to run a speech recognition feature directly on this module, that will then publish the recognized words on the /nicla/audio_recognized
topic. At the moment, VOSK is utilized. Only Arduino version is supported.
VOSK setup
pip install vosk
- Download a VOSK model https://alphacephei.com/vosk/models
- Check the
recognition
arguments in thenicla_receiver.launch
file
Usage
Follow the below steps for enjoying your Arduino Nicla Vision board with ROS!
Run the ROS package
- Launch the package:
$ roslaunch nicla_vision_ros nicla_receiver.launch receiver_ip:="x.x.x.x" connection_type:="tcp/udp" <optional arguments>
- Set the `receiver_ip` with the IP address of your ROS-running machine.
You can get this IP address by executing the following command:
$ ifconfig
and taking the "inet" address under the "enp" voice.
- Set the socket type to be used, either TCP or UDP (`connection_type:="tcp"` or `"udp"`).
Furthermore, using the `<optional arguments>`, you can decide:
- which sensor to be streamed in ROS
(e.g. `enable_imu:=true enable_range:=true enable_audio:=true enable_audio_stamped:=false enable_camera_compressed:=true enable_camera_raw:=true`), and
- on which socket port (default `receiver_port:=8002`).
Once you run it, you will be ready for receiving the sensors data
Simulated board
- For simulating the Arduino Nicla Vision in Gazebo and Rviz:
$ roslaunch nicla_vision_ros nicla_sim.launch <optional arguments>
Using the `<optional arguments>`, you can decide if to run the simulation in Gazebo or in Rviz, and which sensor to simulate (everything set to true as default). ![Alt Text](assets/nicla_rviz.jpg)
Optional Micropython Version
Note this version is not supported and maintained anymore, we defitively moved to the Arduino version.
If you want to use the micropython driver of the nicla (after setup it accordingly), simply run the nicla_receiver.launch
with driver_version:=micropython
as argument.
Video Demonstration
https://github.com/ADVRHumanoids/nicla_vision_ros/assets/26459008/a3eaf921-02ea-4482-80a0-5830a338eb74
Package List
Here some useful links:
License
Distributed under the Apache-2.0 License. See LICENSE for more information.
Citation
:raised_hands: If you use this work or take inspiration from it, please cite (to be published):
@inproceedings {DelBianco2024,
author = {Del Bianco, Edoardo and Torielli, Davide and Rollo, Federico and Gasperini, Damiano and Laurenzi, Arturo and Baccelliere, Lorenzo and Muratore, Luca and Roveri, Marco and Tsagarakis, Nikos G.},
booktitle={2024 IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids)},
title = {A High-Force Gripper with Embedded Multimodal Sensing for Powerful and Perception Driven Grasping},
year={2024},
volume={},
number={},
pages={},
doi={}
}
Changelog for package nicla_vision_ros
1.0.2 (2024-10-11)
- citation
- print fix
- tof on image util
- Updating img compressed cv2_to_compressed_imgmsg
- Update README.md
- Contributors: Davide Torielli, damigas
1.0.1 (2024-07-19)
- license, contrib, and linting
- Arduino version is the default
- speech recognizer
- capturing full queues
- merging from devel - easy part
- rviz image updated
- python3
- Fix comments
- Support Nicla Disconnection on TCP
- Fix ros timestamp tcp
- Fix time and udp check size
- fix python3
- fix ros time
- Update README.md
- Update README.md
- Update README.md
- Add img
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Merge branch 'devel' plus last installs
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Create LICENSE
- Update README.md
- modifying license
- modifying license
- Update README.md
- Update README.md
- Fixing nicla sim launch
- Fixing nicla sim launch
- Fixing nicla sim launch
- Update README.md
- rename image
- Adding assets
- Adding audio player
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- prova README.md
- prova README.md
- UDP/TCP ready
- New main4Nicla
- New main4Nicla
- Fixing CMakeList
- solved bad install and ros2 in another repo
- Merge branch 'devel' of https://github.com/ADVRHumanoids/niclabox into devel Damiano tcp updates with tori packaging files shifts
- organizing python package as ROS1 wants
- simulated nicla
- Adding TCP server and new main4Nicla
- nicla tcp main
- tcp option server side
- Fix launch
- ROS2 nicla receiver
- some package polish
- correct bad bug
- pub camera info filled message
- typo fix
- imu implemented
- Merge branch 'devel' of https://github.com/ADVRHumanoids/niclabox into devel
- receiver developments
- new main4nicla
- to test yet
- ordered things
- trials also with ros version
- fixes
- dev toward classes
- Merge branch 'master' into devel
- Backup main client
- Integrating mic
- Update nicla_macro.urdf.xacro
- Fix urdf laser
- Fix urdf and add deprojection
- Fix urdf with camera parameters
- Fix urdf camera parameters
- Adding calib data in folder
- Changing readme
- Adding img_raw and camera info to server
- camera urdf and gazebo plugin
- gdb and centauro wlan ip setup
- not blocking receiver
- I dont remember about these socket options do
- connect to wlan with static ip
- Typo fixed
- repo address changed to leobotics
- set picture quality to avoid enomem
- Finalised quality to get no error
- CV2 window init outside of loop
- Set image quality at maximum for no error
- Updated readme
- Added license
- Updated comments
- Update README.md
- Removed IP and reordered
- Update README.md
- renamed and updated to remove IP
- Removed network SSID and Password
- readme example
- modified pkg.xml
- updated licenses
- Create LICENSE
- updated niclabox_server node
- created launch file for niclabox_server
- adding dependencies in cmake and pkg.xml
- removing useless script
- updated printing server
- Create README.md
- ROS node created
- Error handling
- first commit
- Contributors: Damiano Gasperini, Davide Torielli, Edoardo Del Bianco, Federico Rollo, damigas, edodelbianco
Wiki Tutorials
Package Dependencies
Deps | Name |
---|---|
catkin | |
xacro | |
gazebo_ros | |
robot_state_publisher | |
rviz | |
rospy | |
sensor_msgs | |
audio_common_msgs | |
cv_bridge |
System Dependencies
Dependant Packages
Launch files
- launch/nicla_receiver.launch
-
- nicla_name [default: nicla]
- receiver_ip
- receiver_port [default: 8002]
- connection_type
- driver_version [default: arduino]
- enable_range [default: true]
- enable_camera_raw [default: false]
- enable_camera_compressed [default: true]
- enable_audio [default: true]
- enable_audio_stamped [default: false]
- enable_audio_recognition_vosk [default: false] — Enable speech recognition with VOSK (need to be installed)
- audio_recognition_model_path [default: ]
- audio_recognition_listen_seconds [default: 2] — The speech recognition will process audio blocks of this duration
- audio_recognition_grammar [default: []] — VOSK format for grammar, [] to use default model graph or as JSON array of strings like: ["open", "bottle", "cup", "[unk]"]
- audio_recognition_wave_output_filename [default: ] — store detected audio in subsequent files. Absolute path may be included. If not, wav files will be store in .ros folder Mostly for debug purposes, leave empty for not storing
- enable_imu [default: true]
- launch/nicla_tof_on_image.launch
-
- nicla_name [default: nicla]
- nicla_cam_topic [default: /nicla/camera/image_raw/compressed]
- nicla_cam_compressed [default: true] — are msg from 'nicla_cam_topic' compressed images?
- nicla_range_topic [default: /nicla/tof]
- img_out_topic [default: /nicla/camera_with_tof/image_raw/compressed]
- launch/nicla_sim.launch
-
- gazebo [default: true]
- rviz [default: true]
- rviz_config [default: $(find nicla_vision_ros)/config/rviz/nicla.rviz]
- nicla_camera [default: true]
- nicla_tof [default: true]
- nicla_visualize_tof [default: true]
- nicla_mic [default: true]
- nicla_imu [default: true]