Multimodal Large Language Models for Real-Time Situated Reasoning

TL;DR

In this work, we show that Large Language Models can go beyond just detecting human values in text and images: they can use these to take action, while at the same time reasoning on the preferences, comfort and safety of the people and pets involved… all of this in real-time! As an example, we observed how an LLM would control a simple vacuum cleaning robot.

To achieve this, we combined an LLM with a TurtleBot 4 platform simulating a smart vacuum cleaning robot in a home. The robot periodically rotates and sends images from its camera to the LLM, which evaluates the environment through a vision pipeline, and determines whether it is appropriate to initiate cleaning. The graphical user interface shows the reasoning process. While this is just a toy example, in the future we plan to use LLMs as the reasoning engine controlling the robot’s behaviours. The video below shows the experiment in action in the Homelab.

Abstract

In this work, we explore how multimodal large language models can support real-time context- and value-aware decision-making. To do so, we combine the GPT-4o language model with a TurtleBot 4 platform simulating a smart vacuum cleaning robot in a home. The model evaluates the environment through vision input and determines whether it is appropriate to initiate cleaning. The system highlights the ability of these models to reason about domestic activities, social norms, and user preferences and take nuanced decisions aligned with the values of the people involved, such as cleanliness, comfort, and safety. We demonstrate the system in a realistic home environment, showing its ability to infer context and values from limited visual input. Our results highlight the promise of multimodal large language models in enhancing robotic autonomy and situational awareness, while also underscoring challenges related to consistency, bias, and real-time performance.

@inproceedings{abbo2026multimodal-large-language,
author = {Abbo, Giulio Antonio and Lenaerts, Senne and Belpaeme, Tony},
title = {Multimodal Large Language Models for Real-Time Situated Reasoning},
year = {2026},
isbn = {9798400723216},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3776734.3796242},
doi = {10.1145/3776734.3796242},
booktitle = {Companion Proceedings of the 21st ACM/IEEE International Conference on Human-Robot Interaction},
pages = {1161–1163},
numpages = {3},
keywords = {Embodied, Home Environment, Human-Robot Interaction, LLM, Reasoning, Social Robot, TurtleBot, VLM, Vacuum Cleaner, Vision},
location = {Edinburgh, Scotland, UK},
series = {HRI Companion '26}
}