I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots
Giulio Antonio Abbo and Tony BelpaemeTL;DR
This work was carried out in November 2023 as soon as OpenAI GPT models with image input were released. Using a Furhat robot, we explored what can be achieved with these techniques and how to interleave images in the LLM’s prompt.
PUBLICATIONS IN THIS RESEARCH LINE
Embodied Awareness
Check out this blog post for an example of how to implement vision-enabled dialogue with a robot.
Abstract
In the rapidly evolving landscape of human-robot interaction, the integration of vision capabilities into conversational agents stands as a crucial advancement. This paper presents a ready-to-use implementation of a dialogue manager that leverages the latest progress in Large Language Models (e.g., GPT-4o mini) to enhance the traditional text-based prompts with real-time visual input. LLMs are used to interpret both textual prompts and visual stimuli, creating a more contextually aware conversational agent. The system's prompt engineering, incorporating dialogue with summarisation of the images, ensures a balance between context preservation and computational efficiency. Six interactions with a Furhat robot powered by this system are reported, illustrating and discussing the results obtained. The system can be customised and is available as a stand-alone application, a Furhat robot implementation, and a ROS2 package.
ACM Reference
Giulio Antonio Abbo and Tony Belpaeme. 2025. I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots. In Proceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction (HRI '25). IEEE Press, 1176–1180.BibTeX
@inproceedings{10.5555/3721488.3721641,
author = {Abbo, Giulio Antonio and Belpaeme, Tony},
title = {I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots},
year = {2025},
publisher = {IEEE Press},
abstract = {In the rapidly evolving landscape of human-robot interaction, the integration of vision capabilities into conversational agents stands as a crucial advancement. This paper presents a ready-to-use implementation of a dialogue manager that leverages the latest progress in Large Language Models (e.g., GPT-4o mini) to enhance the traditional text-based prompts with real-time visual input. LLMs are used to interpret both textual prompts and visual stimuli, creating a more contextually aware conversational agent. The system's prompt engineering, incorporating dialogue with summarisation of the images, ensures a balance between context preservation and computational efficiency. Six interactions with a Furhat robot powered by this system are reported, illustrating and discussing the results obtained. The system can be customised and is available as a stand-alone application, a Furhat robot implementation, and a ROS2 package.},
booktitle = {Proceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction},
pages = {1176–1180},
numpages = {5},
keywords = {conversation, dialogue, hri, large language model, prompt engineering, ros, vision language model},
location = {Melbourne, Australia},
series = {HRI '25}
}