Vision Language Models as Values Detectors
Giulio Antonio Abbo and Tony BelpaemeTL;DR
This work extend the evaluation of LLMs alignment by analysing the responses of models with image input. Compared with humans, the models focus on other aspects, so they are not aligned. However, with proper prompting their answers can be more nuanced than the humans and thus they could be used to reason on values.
PUBLICATIONS IN THIS RESEARCH LINE
Values and Generative AI
Abstract
Large Language Models integrating textual and visual inputs have introduced new possibilities for interpreting complex data. Despite their remarkable ability to generate coherent and contextually relevant text based on visual stimuli, the alignment of these models with human perception in identifying relevant elements in images requires further exploration. This paper investigates the alignment between state-of-the-art LLMs and human annotators in detecting elements of relevance within home environment scenarios. We created a set of twelve images depicting various domestic scenarios and enlisted fourteen annotators to identify the key element in each image. We then compared these human responses with outputs from five different LLMs, including GPT-4o and four LLaVA variants. Our findings reveal a varied degree of alignment, with LLaVA 34B showing the highest performance but still scoring low. However, an analysis of the results highlights the models’ potential to detect value-laden elements in images, suggesting that with improved training and refined prompts, LLMs could enhance applications in social robotics, assistive technologies, and human-computer interaction by providing deeper insights and more contextually relevant responses.
ACM Reference
Giulio Antonio Abbo and Tony Belpaeme. 2025. Vision Language Models as Values Detectors. In Value Engineering in Artificial Intelligence (VALE '24), Springer Nature Switzerland, Cham, 76–86.BibTeX
@InProceedings{10.1007/978-3-031-85463-7_5,
author="Abbo, Giulio Antonio
and Belpaeme, Tony",
editor="Osman, Nardine
and Steels, Luc",
title="Vision Language Models as Values Detectors",
booktitle="Value Engineering in Artificial Intelligence",
year="2025",
publisher="Springer Nature Switzerland",
address="Cham",
pages="76--86",
abstract="Large Language Models integrating textual and visual inputs have introduced new possibilities for interpreting complex data. Despite their remarkable ability to generate coherent and contextually relevant text based on visual stimuli, the alignment of these models with human perception in identifying relevant elements in images requires further exploration. This paper investigates the alignment between state-of-the-art LLMs and human annotators in detecting elements of relevance within home environment scenarios. We created a set of twelve images depicting various domestic scenarios and enlisted fourteen annotators to identify the key element in each image. We then compared these human responses with outputs from five different LLMs, including GPT-4o and four LLaVA variants. Our findings reveal a varied degree of alignment, with LLaVA 34B showing the highest performance but still scoring low. However, an analysis of the results highlights the models' potential to detect value-laden elements in images, suggesting that with improved training and refined prompts, LLMs could enhance applications in social robotics, assistive technologies, and human-computer interaction by providing deeper insights and more contextually relevant responses.",
isbn="978-3-031-85463-7"
}