[](https://opensource.org/licenses/MIT) [](https://www.python.org/downloads/) [](https://docs.ros.org/en/humble/index.html) [](https://discord.gg/B9ZU6qjzND) [](docs/README.zh.md) [](docs/README.ja.md) **The production-grade framework for deploying Physical AI** [**Installation**](#installation) | [**Quick Start**](#quick-start) | [**Documentation**](https://automatika-robotics.github.io/embodied-agents/) | [**Discord**](https://discord.gg/B9ZU6qjzND)

Overview

EmbodiedAgents enables you to create interactive, physical agents that do not just chat, but understand, move, manipulate, and adapt to their environment.

Unlike standard chatbots, this framework provides an orchestration layer for Adaptive Intelligence designed specifically for autonomous systems in dynamic environments.

Core Features

• Production Ready Designed for real-world deployment. Provides a robust orchestration layer that makes deploying Physical AI simple, scalable, and reliable.

• Self-Referential Logic Create agents that are self-aware. Agents can start, stop, or reconfigure their components based on internal or external events. Trivially switch planners based on location, or toggle between cloud and local ML (See: Gödel machines).

• Spatio-Temporal Memory Utilize embodiment primitives like hierarchical spatio-temporal memory and semantic routing. Build arbitrarily complex graphs for agentic information flow. No need to use bloated “GenAI” frameworks on your robot.

• Pure Python, Native ROS2 Define complex asynchronous graphs in standard Python without touching XML launch files. Under the hood, it is pure ROS2—fully compatible with the entire ecosystem of hardware drivers, simulation tools, and visualization suites.

Quick Start

EmbodiedAgents provides a pythonic way to describe node graphs using Sugarcoat.

Copy the following recipe into a python script (e.g., agent.py) to create a VLM-powered agent that can answer questions like “What do you see?”.

from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher

# 1. Define input and output topics
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")

# 2. Define a model client (e.g., Qwen via Ollama)
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)

# 3. Define a VLM component
# A component represents a node with specific functionality
vlm = VLM(
    inputs=[text0, image0],
    outputs=[text1],
    model_client=qwen_client,
    trigger=text0,
    component_name="vqa"
)

# 4. Set the prompt template
vlm.set_topic_prompt(text0, template="""You are an amazing and funny robot.
    Answer the following about this image: {{ text0 }}"""
)

# 5. Launch the agent
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()

Note: Check out the Quick Start Guide or dive into Example Recipes for more details.

Complex Component Graphs

The quickstart example above is just an amuse-bouche of what is possible with EmbodiedAgents. We can create arbitrarily sophisticated component graphs and configure the system to change or reconfigure itself based on events, both internal or external to the system. Check out the code for the following agent here.

Changelog for package automatika_embodied_agents

0.5.1 (2026-02-16)

• (feature) Adds a record_video action to the Vision component
• Takes specific input topic name and duration to record video
• (feature) Adds a take_picture action to the vision component
• (feature) Adds arbitrary action execution to router
• (feature) Adds handling of topic and action lists in semantic router
• (refactor) Makes action methods in Vision component use extra_callback to avoid starvation
• (refactor) Increases default max new tokens param for llm/vlm configs
• (refactor) Makes changes for new api of events/actions in sugar
• (fix) Fixes semantic router handling of action routes based on new api
• (fix) Fixes warmup for model components
• (chore) Adds think option for ollama models
• (chore) Adds proper docstrings to exposed methods and classes
• (docs) Updates model definitions in examples
• (docs) Adds new prompt for llms.txt and new tutorial intros
• (docs) Fix incorrect import in docs
• (docs) Updates installation instructions
• Contributors: ahr, mkabtoul

0.5.0 (2026-01-19)

• (docs) Adds supported types table and component runtype in basics
• (docs) Adds VLA tutorial to docs and refactors them
• (docs) Adds tutorials and examples for using events/fallbacks
• (refactor) Enables health broadcast by default
• (feautre) Adds feature for passing additional model clients to components Component can switch model clients based on events
• (feature) Adds external events as component triggers for certain components that can be run in a timed way
• (feature) Adds stt and tts endpoints and generic model wrappers for the generic http client (openai compatible)
• (feature) Enables support for tool calls in generic http client
• (feature) Adds llms.txt in docs
• (docs) Adds explanation of both routing modes to semantic router docs
• (feature) Adds LLM based agentic routing to semantic router
• (feature) Adds timeout for event based termination if the event doesnt happen within max_timesteps
• (feature) Create observation spec form mapping dicts if dataset info.json is not provided
• (feature) Adds publisher for joint state msg
• (feature) Adds parameter to regulate observation sending rate
• (fix) Adds various fixes for running in multiprocessing (vla)
• (feature) Adds dynamically adjusting loop rate to action server (vla)
• (feature) Adds termination logic to the VLA action server
• (feature) Adds capping of actions based on joint limits before publishing
• (feature) Adds action sending in VLA for multiple datatypes
• (feature) Adds a setter for passing an external aggregation function
• (feature) Adds mapping between LeRobot dataset keys and robot urdf and topics
  • Adds warning and errors for mapped keys, topics
  • Adds creation of input for VLA
  • Adds action receiving and aggregation
• (feature) Adds utility for reading joint limits from URDF files/urls
  • Adds mapping utility function in internal joint state class
• (fix) Adds destruction of stream timer if its created for async websocket clients
• (docs) Adds documentaion for lerobot client
• (feature) Adds an async grpc client for lerobot
• (docs) Removes onnxruntime as a mandatory dependancy, its only required for local models
• (feature) Adds LeRobotPolicy wrapper in models Adds utility for extracting features and actions from info.json
• (feature) Implements replacing input topics to handle trigger callbacks
• (feature) Enables None triggers for Server components
• (fix) Fixes unnecessary logging causing error before component activation
• (fix) Fixes deserialization of map layers and routes in multiprocess run
• Contributors: ahr, mkabtoul

0.4.3 (2025-11-07)

• (docs) Adds instructions for using the dynamic web UI
• (chore) Removes tiny web client
• (fix) Removes StreamingString as input option for LLM/MLLM
• (feature) Adds logging output from StreamingString in the UI
• (chore) Adds warning for using templates with StreamingString
• (chore) Adds alias VLM for MLLM component
• (fix) Adds detections as handled output in mllm component
• (feature) Adds callback for points of interest msg
• (feature) Adds ui callback for rgbd images
• (docs) Updates docs for planning model example
• (fix) Fixes publishing images as part of detection msgs
• (refactor) Updates callbacks for video and rgbd type messages
• (feature) Adds handling of additional types from other sugar derived packages in agent's components
• (feature) Adds UI element definitions for custom types
• (feature) Adds ui callbacks for Detections and DetectionsMultiSource
• (feature) Adds utility for drawing images with bounding boxes
• (fix) Fixes passing topic from sugar derived packages to agents

File truncated at 100 lines see the full file