EmbodiedAgents is a production-grade framework, built on top of ROS2, designed to deploy Physical AI on real world robots. It enables you to create interactive, physical agents that do not just chat, but understand, move, manipulate, and adapt to their environment.

• Production Ready Physical Agents: Designed to be used with autonomous robot systems that operate in real world dynamic environments. EmbodiedAgents makes it simple to create systems that make use of Physical AI. It provides an orchestration layer for Adaptive Intelligence.
• Self-referential and Event Driven: An agent created with EmbodiedAgents can start, stop or reconfigure its own components based on internal and external events. For example, an agent can change the ML model for planning based on its location on the map or input from the vision model. EmbodiedAgents makes it simple to create agents that are self-referential Gödel machines.
• Semantic Memory: Integrates vector databases, semantic routing and other supporting components to quickly build arbitrarily complex graphs for agentic information flow. No need to utilize bloated “GenAI” frameworks on your robot.
• Pure Python, Native ROS2: Define complex asynchronous graphs in standard Python without touching XML launch files. Yet, underneath, it is pure ROS2 compatible with the entire ecosystem of hardware drivers, simulation tools, and visualization suites.

Join our Discord 👾

Checkout Installation Instructions 🛠️

Get started with the Quickstart Guide 🚀

Get familiar with Basic Concepts 📚

Dive right in with Example Recipes ✨

Installation 🛠️

Install a model serving platform

The core of EmbodiedAgents is agnostic to model serving platforms. It supports Ollama, RoboML and all platforms or cloud provider with an OpenAI compatible API (e.g. vLLM, lmdeploy etc.). For VLA models EmbodiedAgents supports policies severed on the Async Inference server from LeRobot. Please install either of these by following the instructions provided by respective projects. Support for new platforms is being continuously added. If you would like to support a particular platform, please open an issue/PR.

Install EmbodiedAgents (Ubuntu)

For ROS versions >= humble, you can install EmbodiedAgents with your package manager. For example on Ubuntu:

sudo apt install ros-$ROS_DISTRO-automatika-embodied-agents

Alternatively, grab your favorite deb package from the release page and install it as follows:

sudo dpkg -i ros-$ROS_DISTRO-automatica-embodied-agents_$version$DISTRO_$ARCHITECTURE.deb

If the attrs version from your package manager is < 23.2, install it using pip as follows:

pip install 'attrs>=23.2.0'

Install EmbodiedAgents from source

Get Dependencies

Install python dependencies

pip install numpy opencv-python-headless 'attrs>=23.2.0' jinja2 httpx setproctitle msgpack msgpack-numpy platformdirs tqdm websockets

Download Sugarcoat🍬

git clone https://github.com/automatika-robotics/sugarcoat

Install EmbodiedAgents

git clone https://github.com/automatika-robotics/embodied-agents.git
cd ..
colcon build
source install/setup.bash
python your_script.py

Quick Start 🚀

Unlike other ROS package, EmbodiedAgents provides a pure pythonic way of describing the node graph using Sugarcoat🍬. Copy the following recipe in a python script and run it.

```python from agents.clients.ollama import OllamaClient from agents.components import VLM from agents.models import OllamaModel from agents.ros import Topic, Launcher

Define input and output topics (pay attention to msg_type)

text0 = Topic(name=”text0”, msg_type=”String”) image0 = Topic(name=”image_raw”, msg_type=”Image”) text1 = Topic(name=”text1”, msg_type=”String”)

Define a model client (working with Ollama in this case)

OllamaModel is a generic wrapper for all Ollama models

llava = OllamaModel(name=”llava”, checkpoint=”llava:latest”) llava_client = OllamaClient(llava)

Define a VLM component (A component represents a node with a particular functionality)

mllm = VLM( inputs=[text0, image0], outputs=[text1], model_client=llava_client, trigger=text0, component_name=”vqa” )

File truncated at 100 lines see the full file

Changelog for package automatika_embodied_agents

0.5.0 (2026-01-19)

• (docs) Adds supported types table and component runtype in basics
• (docs) Adds VLA tutorial to docs and refactors them
• (docs) Adds tutorials and examples for using events/fallbacks
• (refactor) Enables health broadcast by default
• (feautre) Adds feature for passing additional model clients to components Component can switch model clients based on events
• (feature) Adds external events as component triggers for certain components that can be run in a timed way
• (feature) Adds stt and tts endpoints and generic model wrappers for the generic http client (openai compatible)
• (feature) Enables support for tool calls in generic http client
• (feature) Adds llms.txt in docs
• (docs) Adds explanation of both routing modes to semantic router docs
• (feature) Adds LLM based agentic routing to semantic router
• (feature) Adds timeout for event based termination if the event doesnt happen within max_timesteps
• (feature) Create observation spec form mapping dicts if dataset info.json is not provided
• (feature) Adds publisher for joint state msg
• (feature) Adds parameter to regulate observation sending rate
• (fix) Adds various fixes for running in multiprocessing (vla)
• (feature) Adds dynamically adjusting loop rate to action server (vla)
• (feature) Adds termination logic to the VLA action server
• (feature) Adds capping of actions based on joint limits before publishing
• (feature) Adds action sending in VLA for multiple datatypes
• (feature) Adds a setter for passing an external aggregation function
• (feature) Adds mapping between LeRobot dataset keys and robot urdf and topics
  • Adds warning and errors for mapped keys, topics
  • Adds creation of input for VLA
  • Adds action receiving and aggregation
• (feature) Adds utility for reading joint limits from URDF files/urls
  • Adds mapping utility function in internal joint state class
• (fix) Adds destruction of stream timer if its created for async websocket clients
• (docs) Adds documentaion for lerobot client
• (feature) Adds an async grpc client for lerobot
• (docs) Removes onnxruntime as a mandatory dependancy, its only required for local models
• (feature) Adds LeRobotPolicy wrapper in models Adds utility for extracting features and actions from info.json
• (feature) Implements replacing input topics to handle trigger callbacks
• (feature) Enables None triggers for Server components
• (fix) Fixes unnecessary logging causing error before component activation
• (fix) Fixes deserialization of map layers and routes in multiprocess run
• Contributors: ahr, mkabtoul

0.4.3 (2025-11-07)

• (docs) Adds instructions for using the dynamic web UI
• (chore) Removes tiny web client
• (fix) Removes StreamingString as input option for LLM/MLLM
• (feature) Adds logging output from StreamingString in the UI
• (chore) Adds warning for using templates with StreamingString
• (chore) Adds alias VLM for MLLM component
• (fix) Adds detections as handled output in mllm component
• (feature) Adds callback for points of interest msg
• (feature) Adds ui callback for rgbd images
• (docs) Updates docs for planning model example
• (fix) Fixes publishing images as part of detection msgs
• (refactor) Updates callbacks for video and rgbd type messages
• (feature) Adds handling of additional types from other sugar derived packages in agent's components
• (feature) Adds UI element definitions for custom types
• (feature) Adds ui callbacks for Detections and DetectionsMultiSource
• (feature) Adds utility for drawing images with bounding boxes
• (fix) Fixes passing topic from sugar derived packages to agents components
• (feature) Adds Detection2D as allowed input in map encoding component
• (feature) Adds callback for Detections2D and their use in llm/mllm components
  • Streamlines names of detection msgs
  • Streamlines names of tracking msgs
• (fix) Gets raw msg data in execution step to avoid calling get_output twice
• (chore) Adds websockets as an explicit dependency
• Contributors: ahr, mkabtoul

0.4.2 (2025-09-03)

• (feature) Adds udp streaming to IP:PORT as an option to TextToStream component when play_on_device is enabled
• (docs) Updates docs to use new web based client
• (feature) Adds processing of audio messages in web client
• (chore) Removes chainlit based client
• (feature) Adds a custom webclient to replace chainlit
• (feature) Adds persistent ros node in web client for async stream handling
• (feature) Adds warning when not using streaming string msg_type with

File truncated at 100 lines see the full file