# I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots

- Authors: Giulio Antonio Abbo and Tony Belpaeme
- Date: 2025-03-04
- Type: conference paper
- Venue: Proceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction (HRI '25 (LBR))
- DOI: http://dx.doi.org/10.1109/HRI61500.2025.10973830
- Code: https://github.com/giubots/vision-enabled-dialogue
- Supplementary material: https://doi.org/10.5281/zenodo.10127383
- ResearchGate: https://www.researchgate.net/publication/390693470_I_Was_Blind_but_Now_I_See_Implementing_Vision-Enabled_Dialogue_in_Social_Robots
- PDF: https://giubots.net/publications/was-blind-but/was-blind-but.pdf
- Research line: [Embodied Awareness](https://giubots.net/research/embodied-awareness/)
- URL: https://giubots.net/publications/was-blind-but/


## TL;DR

This work was carried out in November 2023 as soon as OpenAI GPT models with image input were released. Using a Furhat robot, we explored what can be achieved with these techniques and how to interleave images in the LLM’s prompt.


This work was carried out in November 2023 as soon as OpenAI GPT models with image input were released. Using a Furhat robot, we explored what can be achieved with these techniques and how to interleave images in the LLM's prompt.

[YouTube video](https://www.youtube.com/watch?v=ihl3zNr2H3E)

Check out [this blog post](posts/vision-enabled-dialogue) for an example of how to implement vision-enabled dialogue with a robot.

## Abstract

In the rapidly evolving landscape of human-robot interaction, the integration of vision capabilities into conversational agents stands as a crucial advancement. This paper presents a ready-to-use implementation of a dialogue manager that leverages the latest progress in Large Language Models (e.g., GPT-4o mini) to enhance the traditional text-based prompts with real-time visual input. LLMs are used to interpret both textual prompts and visual stimuli, creating a more contextually aware conversational agent. The system's prompt engineering, incorporating dialogue with summarisation of the images, ensures a balance between context preservation and computational efficiency. Six interactions with a Furhat robot powered by this system are reported, illustrating and discussing the results obtained. The system can be customised and is available as a stand-alone application, a Furhat robot implementation, and a ROS2 package.

## How to cite

Giulio Antonio Abbo and Tony Belpaeme. 2025. I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots. In Proceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction (HRI '25). IEEE Press, 1176–1180.

```bibtex
@inproceedings{10.5555/3721488.3721641,
author = {Abbo, Giulio Antonio and Belpaeme, Tony},
title = {I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots},
year = {2025},
publisher = {IEEE Press},
abstract = {In the rapidly evolving landscape of human-robot interaction, the integration of vision capabilities into conversational agents stands as a crucial advancement. This paper presents a ready-to-use implementation of a dialogue manager that leverages the latest progress in Large Language Models (e.g., GPT-4o mini) to enhance the traditional text-based prompts with real-time visual input. LLMs are used to interpret both textual prompts and visual stimuli, creating a more contextually aware conversational agent. The system's prompt engineering, incorporating dialogue with summarisation of the images, ensures a balance between context preservation and computational efficiency. Six interactions with a Furhat robot powered by this system are reported, illustrating and discussing the results obtained. The system can be customised and is available as a stand-alone application, a Furhat robot implementation, and a ROS2 package.},
booktitle = {Proceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction},
pages = {1176–1180},
numpages = {5},
keywords = {conversation, dialogue, hri, large language model, prompt engineering, ros, vision language model},
location = {Melbourne, Australia},
series = {HRI '25}
}
```