dialogflow_task_executive

A ROS package for Task execution with Dialogflow

System structure

Installation to your new robot

Create a Dialogflow account

Go to Dialogflow website and create an account.

If you are a JSK lab member, ask s-kitagawa or k-okada about JSK dialogflow account.

Create new Agent on Dialogflow

Go to console on Dialogflow Dialogflow and create new agent.

Create new Service account key

Go to this page and create service account key.

The page is as shown below and press blue button to create the key.

You can download the key as JSON file in the age and save the JSON file in your robot.

Set configuration

Use environmental variables

We need to set two environment variables: GOOGLE_APPLICATION_CREDENTIALS and DIALOGFLOW_PROJECT_ID.

• GOOGLE_APPLICATION_CREDENTIALS
  • Path to Service account key JSON file (i.e. /etc/ros/hogehoge.json)
• DIALOGFLOW_PROJECT_ID
  • Dialogflow project ID (i.e. pr2-hogehoge)

Use launch file arguments

Or we can use credential and project_id arguments of ` dialogflow_task_executive.launch` file

roslaunch dialogflow_task_executive dialogflow_task_executive.launch credential:=/etc/ros/hogehoge.json project_id:=pr2-hogehoge

If you have project_id in both your credential file and rospram (or argument of dialogflow_task_executive.launch), the project_id in both your credential file becomes effective. To use project_id in rosparam, use ~override_project_id option.

How to register new task in Dialogflow

Create new app_manager app

First, you need to create your task program to app_manager app. Your app is registered and controled with tag <package name>/<app name> (i.e. your_package/your_demo).

Please read app_manager for more detailed information.

Create new Intent in Dialogflow

For your new task, create new Intent as below.

Action section, you can set full name (<package name>/<app name>), app name or camel-cased name of your app_manager app. The Full name (<package name>/<app name>) is recommended to avoid name confliction.

If your app is registered as your_package/your_demo, you need to set your_package/your_demo, your_demo or YourDemo in Action section.

(i.e. App name: detect_cans_in_fridge201202/pick_object -> Dialogflow Action: detect_cans_in_fridge201202/pick_object, pick_object or PickObject)

In order to fulfill other forms, please read dialogflow doc for more detailed information abount Intent.

Usage

Launch jsk_robot_startup/lifelog/app_manager.launch

roslaunch jsk_robot_startup app_manager.launch

Note: You should skip this If you are using PR2 and Fetch in JSK lab because this launch is running by default.

Launch dialogflow_task_executive/launch/dialogflow_task_executive.launch

roslaunch dialogflow_task_executive dialogflow_task_executive.launch

Note: You can run this launch by sudo initctl jsk-dialog start on PR2 and Fetch in JSK lab.

Ask your robot to do the task

Your robot will execute the task from dialogflow.

Upstart example

Edit a config file below and save it as /etc/init/jsk-dialog.conf

You can control dialogflow_task_executive service by sudo initctl start jsk-dialog.

See config/jsk-dialog.conf

# make sure that robot is running
start on robot-is-up
stop on robot-is-down

respawn
console log

# set your own env parameters
env USERNAME=<user name>
env ROS_ENV_LOADER=<ROS env loader path>
env GOOGLE_APPLICATION_CREDENTIALS=<Google service account key path> 
env DIALOGFLOW_PROJECT_ID=<Dialogflow project ID>

pre-start script
  exec >"/tmp/ros_run_id"
  echo "ROS_IP=$(ip -o route get 8.8.8.8 | awk '{print $7;}')"
end script

script
  exec su $USERNAME -c ". /tmp/ros_run_id; $ROS_ENV_LOADER roslaunch dialogflow_task_executive dialogflow_task_executive.launch run_app_manager:=false --screen --wait"
end script

Sample programs for JSK users

1. Open Dialogflow and setup Intents and Entities.

2. Download google credentials file. You can download from Google Drive link.

3. Start sample code

roslaunch dialogflow_task_executive demo.launch google_cloud_credentials_json:=${HOME}/Downloads/eternal-byte-236613-4bc6962824d1.json

Author

Yuki Furuta furushchev@jsk.imi.i.u-tokyo.ac.jp

Shingo Kitagawa s-kitagawa@jsk.imi.i.u-tokyo.ac.jp

License

BSD

Changelog for package dialogflow_task_executive

2.1.28 (2023-07-24)

2.1.27 (2023-06-24)

• fix package.xml/CMakeLists.txt to supress catkin_lint errors (#479)
• Contributors: Kei Okada

2.1.26 (2023-06-14)

• add LICENSE files (#476)
• Contributors: Kei Okada

2.1.25 (2023-06-08)

• Add option to use project_id from json, instead of credentials (#460)
• add test to check if ros node is loadable (#463)
  • install python files under CATKIN_PACKAGE_BIN_DESTINATION
  • dialogflow_task_executive: add missing dependencies to package.xml
  • dialogflow_task_executive: pip -i only works with 64 bit (x86_64 and aarch64), other archtecture skips rospy_node.test
  • dialogflow_task_executive/CMakeLists.txt: test/test_rospy_node.test depends on generate_messages
  • dialogflow_task_executive/requirements.txt.indigo: specify grpcio for indigo
  • add catkin_install_python for test, it is also change installed directory from BIN to SHARE, because we want to have same directory structure between devel and install
  • add test to check if ros node is loadable If we use python2 PYTHON_INTERPRETER on 20.04, python2 fails to load rospy in /opt/ros/noetic, because rospy moduels are alraedy updated. If we use python3 PYTHON_INTERPRETER on 18.04, python3 can load rospy in /opt/ros/melodic by chance. c.f. https://github.com/jsk-ros-pkg/jsk_3rdparty/pull/367
• [dialogflow_task_executive] chmod -x because of catkin_virtualenv (#449)
• [dialogflow_task_executive] not use map for python3 in task_executive (#447)
• [dialogflow_task_executive] fix typo in dialogflow_client.py (#445)
  • chmod -x because of catkin_virtualenv
  • not use map for python3 in task_executive
  • fix typo in dialogflow_client.py
• [dialogflow_task_executive] add sample apps (#293)
• [dialogflow_task_executive] add speech_to_text_other topic name in speech_to_text_mux (#302)
• [CI] build catkin_virtualenv packages in indigo. (#419)
• [dialogflow_task_executive] check if ROS_PYTHON_VERSION is set to support indigo in dialogflow_client (#408)
• [#405 and #410] Fix CI (#415)
• [dialogflow_task_executive] Support noetic (#362)
• Fix GithubAction (#386)
• [dialogflow_task_executive] Enable aarch64 (#364)
• GithubAction: add test for aarch64(melodic) / indigo (arm64) (#365)
• [dialogflow_task_executive] set optenv as default false (#368)
• Explicit python interpreter in catkin_virtualenv (#367)
• [dialogflow_task_executive] add comments in DialogResponse.msg (#366)
• [dialogflow_task_executive] adding some args in dialogflow_ros.launch (#361)
• [dialogflow_task_executive] fix warn bug when got unknown action (#363)
• fix type on dialogflow_task_executive/requirements.txt.indigo (#355)
• [dialogflow_task_executive] add requirements.txt in .gitignore (#353)
• [dialogflow_task_executive] add doc in launch (#349)
• [dialogflow_task_executive]separate dialogflow API client and app execution from dialogflow_task_executive (#343)
• dialogflow_task_executive: re-enable idnigo (#342)
• dialogflow_task_executive: more depends and udpate README.md (#334)
• [dialogflow_task_executive] Fix bugs when subscribing /text (std_msgs/String) topic (#328)
• [dialogflow_task_executive] load dialogflow project_id from credentials (#319)
• [dialogflow_task_executive] Fix merging #317 (#318)
• Support dialogflow hotword yaml (#307)
• add sample code of dialogflow (#317)
• Contributors: Kei Okada, Koki Shinjo, Naoto Tsukamoto, Naoya Yamaguchi, Shingo Kitagawa, Yoshiki Obinata, Iory Yanokura

2.1.24 (2021-07-26)

2.1.23 (2021-07-21)

2.1.22 (2021-06-10)

• [dialogflow_task_executive] Change mux namespace for speech_to_textf( #221)
• Contributors: Naoya Yamaguchi

2.1.21 (2020-08-19)

2.1.20 (2020-08-07)

• [dialogflow_task_executive] accepct full name of app name for dialogflow action (#208)
  • update readme to accept dialogflow action name
  • use whole app_name for action
• Contributors: Kei Okada, Shingo Kitagawa

2.1.19 (2020-07-21)

2.1.18 (2020-07-20)

2.1.17 (2020-04-16)

2.1.16 (2020-04-16)

2.1.15 (2019-12-12)

• add url in dialogflow_task_executive (#181)
• Contributors: Shingo Kitagawa

2.1.14 (2019-11-21)

• [dialogflow_task_executive] add dialogflow_task_executive (#165)
  • add std_msgs in package.xml (#177)
  • update system fig/img
  • add jsk-dialog.conf example
  • add upstart example
• Contributors: Shingo Kitagawa