EmbodiedAgents is a fully-loaded framework, written in pure ROS2, for creating interactive physical agents that can understand, remember, and act upon contextual information from their environment.

• Production Ready Physical Agents: Designed to be used with autonomous robot systems that operate in real world dynamic environments. EmbodiedAgents makes it simple to create systems that make use of Physical AI.
• Intuitive API: Simple pythonic API to utilize local or cloud based ML models (specifically Multimodal LLMs and other transformer based architectures) on robots.
• Semantic Memory: Integrates vector databases, semantic routing and other supporting components to quickly build arbitrarily complex graphs for agentic information flow. No need to utilize bloated “GenAI” frameworks on your robot.
• Made in ROS2: Utilizes ROS2 as the underlying distributed communications backbone. Theoretically, all devices that provide a ROS2 package can be utilized to send data to ML models, with callbacks implemented for most commonly used data types and infinite extensibility.

Checkout Installation Instructions 🛠️

Get started with the Quickstart Guide 🚀

Get familiar with Basic Concepts 📚

Dive right in with Examples ✨

Installation 🛠️

Install a model serving platform

The core of EmbodiedAgents is agnostic to model serving platforms. It currently supports Ollama, RoboML and any platform or cloud provider with an OpenAI compatible API (e.g. vLLM, lmdeploy etc.). Please install either of these by following the instructions provided by respective projects. Support for new platforms is being continuously added. If you would like to support a particular platform, please open an issue/PR.

Install EmbodiedAgents (Ubuntu)

For ROS versions >= humble, you can install EmbodiedAgents with your package manager. For example on Ubuntu:

sudo apt install ros-$ROS_DISTRO-automatika-embodied-agents

Alternatively, grab your favorite deb package from the release page and install it as follows:

sudo dpkg -i ros-$ROS_DISTRO-automatica-embodied-agents_$version$DISTRO_$ARCHITECTURE.deb

If the attrs version from your package manager is < 23.2, install it using pip as follows:

pip install 'attrs>=23.2.0'

Install EmbodiedAgents from source

Get Dependencies

Install python dependencies

pip install numpy opencv-python-headless 'attrs>=23.2.0' jinja2 httpx setproctitle msgpack msgpack-numpy platformdirs tqdm

Download Sugarcoat🍬

git clone https://github.com/automatika-robotics/sugarcoat

Install EmbodiedAgents

git clone https://github.com/automatika-robotics/embodied-agents.git
cd ..
colcon build
source install/setup.bash
python your_script.py

Quick Start 🚀

Unlike other ROS package, EmbodiedAgents provides a pure pythonic way of describing the node graph using Sugarcoat🍬. Copy the following code in a python script and run it.

```python from agents.clients.ollama import OllamaClient from agents.components import MLLM from agents.models import OllamaModel from agents.ros import Topic, Launcher

Define input and output topics (pay attention to msg_type)

text0 = Topic(name=”text0”, msg_type=”String”) image0 = Topic(name=”image_raw”, msg_type=”Image”) text1 = Topic(name=”text1”, msg_type=”String”)

Define a model client (working with Ollama in this case)

llava = OllamaModel(name=”llava”, checkpoint=”llava:latest”) llava_client = OllamaClient(llava)

Define an MLLM component (A component represents a node with a particular functionality)

mllm = MLLM( inputs=[text0, image0], outputs=[text1], model_client=llava_client, trigger=[text0], component_name=”vqa” )

Additional prompt settings

mllm.set_topic_prompt(text0, template=”"”You are an amazing and funny robot. Answer the following about this image: {{ text0 }}”””

File truncated at 100 lines see the full file