Language models are increasingly capable of reasoning, using tools and coordinating with other agents. But what changes when an agent can’t just generate an answer and has to act in the physical world?

Drawing on several research projects, from sensorimotor perception to the control of rehabilitation robots, I explore three fundamental challenges: giving the agent a representation of its own body from sensory information, translating human intentions into valid robotic actions, and using feedback from the environment to adapt its behaviour.

The talk walks through these challenges with examples from real robotic systems. I’ll cover how multimodal LLMs can infer aspects of the robot’s own embodiment from its sensors, how multi-agent architectures make it possible to separate clinical reasoning from technical control, and how language-guided systems can adapt their actions through closed-loop interaction.

The central idea is that connecting an LLM to a robot goes far beyond giving it access to a tool. It requires representing the system’s capabilities and limitations, building a reliable interface between reasoning and action, and incorporating feedback and supervision mechanisms.

The goal is to share practical lessons from building these systems and discuss what we need to move from eye-catching AI agent demos to systems that can act reliably in the real world.