ROS Index
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
No version for distro ardent showing humble. Known supported distros are highlighted in the buttons above.
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
No version for distro bouncy showing humble. Known supported distros are highlighted in the buttons above.
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
No version for distro crystal showing humble. Known supported distros are highlighted in the buttons above.
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
No version for distro eloquent showing humble. Known supported distros are highlighted in the buttons above.
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
No version for distro dashing showing humble. Known supported distros are highlighted in the buttons above.
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
No version for distro galactic showing humble. Known supported distros are highlighted in the buttons above.
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
No version for distro foxy showing humble. Known supported distros are highlighted in the buttons above.
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
No version for distro iron showing humble. Known supported distros are highlighted in the buttons above.
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
No version for distro lunar showing humble. Known supported distros are highlighted in the buttons above.
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
No version for distro jade showing humble. Known supported distros are highlighted in the buttons above.
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
No version for distro indigo showing humble. Known supported distros are highlighted in the buttons above.
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
No version for distro hydro showing humble. Known supported distros are highlighted in the buttons above.
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
No version for distro kinetic showing humble. Known supported distros are highlighted in the buttons above.
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
No version for distro melodic showing humble. Known supported distros are highlighted in the buttons above.
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.
Recent questions tagged automatika_embodied_agents at Robotics Stack Exchange
No version for distro noetic showing humble. Known supported distros are highlighted in the buttons above.
|
|
automatika_embodied_agents package from automatika_embodied_agents repoautomatika_embodied_agents |
ROS Distro
|
Package Summary
| Version | 0.8.0 |
| License | MIT |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/automatika-robotics/ros-agents.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-10-21 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
agents
Additional Links
Maintainers
- Automatika Robotics
Authors
No additional authors.
**The production-grade framework for deploying Physical AI** [](LICENSE) [](https://emos.automatikarobotics.com/getting-started/installation.html) [](https://discord.gg/B9ZU6qjzND) [**Documentation**](https://emos.automatikarobotics.com) · [**Developer Docs**](https://automatika-robotics.github.io/embodied-agents/) · [**Recipes**](https://emos.automatikarobotics.com/recipes/overview.html) · [**Discord**](https://discord.gg/B9ZU6qjzND)
An SO-101 arm picks oranges in Isaac Sim with a GR00T N1.7 policy, run by the VLA component. An event stops the policy when a VLM watching the camera judges the task done (recipe).
EmbodiedAgents is the intelligence and manipulation layer of EMOS, the Embodied Operating System. An agent is a graph of components, written in one Python file called a recipe: models that see, listen, speak, remember, plan and act, connected by ROS 2 topics and events. Every component is a managed ROS 2 lifecycle node, and the graph can reconfigure itself while it runs.
Components
| Area | Components |
|---|---|
| Language and vision | LLM and VLM answer, call tools, describe and ground what the camera sees, and keep a turn-based chat history. SemanticRouter sends each query to the right component, by meaning, by an LLM or by a decision model. |
| Decisions | DecisionComponent asks a decision model typed questions (yes or no, a choice, a score) and publishes each answer with its probabilities, for events to act on. |
| Perception | Vision detects and tracks objects, and lifts its 2D boxes to metric 3D boxes on depth. MotionDetector finds motion in images or point clouds, ignoring the robot’s own. |
| Speech | SpeechToText, with voice activity detection and a wake word, and TextToSpeech. |
| Manipulation | MoveIt plans collision-free motion, picks and places through MoveIt 2, around the objects the cameras see. VLA runs vision-language-action policies served by LeRobot, such as GR00T, π0.5, SmolVLA and ACT. |
| Memory | Memory keeps a spatio-temporal graph of what the robot saw, where and when (eMEM), with tools to search it. |
| Planning | Cortex plans a task in plain language, with every action in the recipe as a tool, and runs the plan as monitored routines. |
Key Features
- Agents that adapt on the fly. Any component can react to what another one publishes. Let a person walking into view wake up your VLM, or let a decision model call time on a VLA the moment the job is done. If the cloud goes down mid-task, the agent switches to a local model and carries on.
- Classical and learned manipulation. MoveIt plans around what Vision lifts into 3D, and VLA runs learned policies. A decision model or a VLM watching the camera can end a policy’s task the moment it is done.
- Run fully offline. Built-in local models for LLM, VLM, detection, speech-to-text and text-to-speech (llama.cpp, ONNX Runtime and sherpa-onnx), with no server. Suited to edge devices and NVIDIA Jetson.
- Any model server. Ollama, OpenAI-compatible APIs (vLLM, SGLang, llama.cpp, LMDeploy), RoboML and the LeRobot policy server, with Chroma for vector search.
- Planning when you need it. Hand Cortex a task in plain language and it builds a plan from everything your recipe can already do, calling on any component along the way. Each plan runs as a routine you can pause or abort from the web UI.
A Recipe Is a Graph
In this recipe, speech reaches a VLM that answers out loud and tells the memory what it sees. Vision’s 3D boxes become MoveIt’s planning scene, and a decision model watching the camera ends a VLA task through an event. Cortex can join a graph like this one and plan with every component’s actions as its tools, including those of other packages such as Kompass for navigation.
Quick Start
A VLM agent that answers questions about what its camera sees:
from agents.clients.ollama import OllamaClient
from agents.components import VLM
from agents.models import OllamaModel
from agents.ros import Topic, Launcher
text0 = Topic(name="text0", msg_type="String")
image0 = Topic(name="image_raw", msg_type="Image")
text1 = Topic(name="text1", msg_type="String")
qwen_vl = OllamaModel(name="qwen_vl", checkpoint="qwen2.5vl:latest")
qwen_client = OllamaClient(qwen_vl)
vlm = VLM(
inputs=[text0, image0],
outputs=[text1],
model_client=qwen_client,
trigger=text0,
component_name="vqa"
)
launcher = Launcher()
launcher.add_pkg(components=[vlm])
launcher.bringup()
Components are wired with topics and events. Here a detector runs on every frame with a small local model, and the VLM describes the scene only when a person appears:
```python from agents.components import Vision from agents.config import VisionConfig from agents.ros import Event, FixedInput
detections = Topic(name=”detections”, msg_type=”Detections”)
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package automatika_embodied_agents
0.8.0 (2026-10-20)
- (chore) Bumps required min sugarcoat version
- (feature) Adds support for decision models in ollama client and shows model pull progress
- (feature) Adds model download in thread in model base along with model progress
- (fix) Fixes stamping messages correctly in vision processing
- (chore) Removes deprecated map encoding component
- (fix) Adds correct handling of additional types in recipes which include other sugarcoated packages
- (docs) Updates examples with chat history set to true
- (feature) Adds getting and resetting chat history as actions for llm
- (feature) Adds handling cortex planning calls as chat history
- (fix) Serialize pydantic tool calls returned in ollama client
- (feature) Adds summarization and use of chat history in cortex planning loop
- (feature) Adds calculation of history budget from models reported context for supporting clients
- (feature) Adds turn by turn chat history with turn and token budget based trimming
- (fix) Fixes resetting chat history with the reset phrase preserving system prompt
- (fix) Fixes sending images correctly as part of the last message to generic and ollama clients
- (docs) Updates docs for scratch functions
- (feature) Adds a scratch pad for the planner to write custom functions for actions and internal events when none available
- (fix) Fixes passing think option to ollama models
- (chore) Adds decision model example
- (feature) Adds calling component actions for state gathering in decision component
- (refactor) Adds overriding base component method for creating service clients in model components
- (refactor) Simplifies external component tool calling in llm
- (refactor) Lifts calling component actions on other components to model component
- (feature) Adds a component action to get current task
- (feature) Makes cortex monitoring loop use tool calls instead of text response parsing and handled failures and skipped steps correctly
- (feature) Adds DECISION mode in semantic router for using decision models
- (feature) Adds a decision component for calling decision models
- (feature) Adds a decision msg and its sender/callback for one answer from a decision component
- (feature) Adds calling the systemone decision api in the generic http client
- (feature) Adds generic decision model wrapper
- (fix) Fixes deinit in clients to keep the connection open for safe restart
- (fix) Fixes bug in chroma db deinit and creating a collection without a distance function
- (chore) Fixes integration tests and remove the ignore in the CI
- (docs) Adds minor edit in cortex docs
- (fix) Adds a small gil release sleep in vla busy loop when inputs not received
- (fix) Fixes tool call results being sent back to model in llm component
- (fix) Fixes ollama model init for models already on disk are not pulled
- (refactor) Standardizes tool call parsing for OpenAI compatible servers
- (refactor) Adds calling external processors based on the new contract in llm and vla components
- (fix) Fixes using events as triggers in components by returning the correct action return type
- (refator) Replaces dots with hyphens in tool names for cortex
- (feature) Adds plugin tools to be compilable into routines and removes duplicates when sending tools to planner
- (feature) Makes planning actions compilable into a routine
- (refactor) Registers and calls plugins actions via the monitor
- (feature) Adds dynamically adding events/actions during planning step in cortex
- (feature) Adds compilation of plan steps into monitored routines in cortex execution
- (feature) Adds adding headless routines to cortex for being called and monitored as action servers
- (feature) Adds a wait tool in cortex to dwell on actions after calling them so they can take effect
- (fix) Fixes how cortex parses action responses
- (feature) Adds checksum verification to local model downloads that we manage ourselves
- (fix) Strips the scheme if added to the lerobot client host with a warning
- (fix) Fixes correct handling of wss endpoints in roboml websocket client
- (feature) Adds a ca cert option to provide a self signed certificate for trusted self hosted model and db servers
- (feature) Adds proper serialization of key env variable and warning when sending over http
- (feature) Adds issuing a warning for non encrypted external connections when not to loopback addresses
- (feature) Adds registration of sensor plugins in planning prompt and adds proper execution of plugin actions as launcher process methods with kwargs
- (feature) Adds ui callback for trackings
- (feature) Adds ui callback for jointstate
File truncated at 100 lines see the full file
Package Dependencies
| Deps | Name |
|---|---|
| ament_cmake | |
| ament_cmake_python | |
| rosidl_default_generators | |
| rosidl_default_runtime | |
| builtin_interfaces | |
| std_msgs | |
| sensor_msgs | |
| geometry_msgs | |
| automatika_ros_sugar |
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
Services
No service files found
Plugins
No plugins found.