Embodied Awareness
When I started working on the VALAWAI project, I had no idea what the people writing the proposal meant by “awareness”. In this research line, I explore what machine awareness is and how we can improve contextual awareness, especially in social robots.
A Difficult Definition
What was clear from the beginning was that “awareness” was different from “consciousness”, an unpopular term that comes with a far too heavy philosophical and scientific baggage for a PhD student. But I could not wrap my head around the difference between the two terms. In addition, the VALAWAI consortium made it clear that we should always talk about being aware of something. So in the end, the imaginary line that I ended up tracing to distinguish the two is the of itself: saying that someone is aware – without specifying what they are aware of – for me was a synonym of being conscious; everything else was “just awareness”.
Now that consciousness was out of the picture, another problem came up. I will provide an example. If I say that I am aware of the consequences, this means that I know what the consequences are. But surely awareness should be more than mere knowledge, at least that is what my gut feelings said.
Not wanting to fall into the rabbit hole that is the definition of “knowledge” and what it means to know something, and keeping myself well away from the risk of ending up having to define what “understanding” is, I decided to settle on “having access to information” as one of the requirements for machine awareness. From this, other requirements followed, giving a better idea of what the necessary aspects of an aware, embodied AI are.
Neurosciences to the Rescue
Luckily for me, the VALAWAI project proposal pretty much defined awareness as something that can be achieved through the implementation of the Dehaene-Changeux computational model for consciousness awareness, which in turn is based on Baar’s Global Workspace Theory. Wikipedia does a far better job at summarising it.
PUBLICATION
Embodied Awareness
I argue that the following aspects are necessary for an aware embodied AI: having access to relevant information, also via memory recall; directing the system’s attention appropriately, filtering out distractions; and being able to integrate information into a coherent model. An aware system should respond based on that model, adapting its internal state or actions in the environment while maintaining coherence in its decisions during the task execution and over time. Finally, its decisions should be explainable and supported by verifiable outputs. Change my mind.
PUBLICATION
VLMs for Contextual Awareness
This work was carried out in November 2023 as soon as OpenAI GPT models with image input were released. Using a Furhat robot, we explored what can be achieved with these techniques and how to interleave images in the LLM’s prompt.
PUBLICATION
Multi-Party Open-Ended Conversation With a Social Robot
In this work we explore wether an LLM can guide a multi-party spoken interaction. To track speakers, in our setting face recognition performed best; voice recognition was unreliable due to frequent short utterances and overlapping speech. The response and its addressee were generated by the LLM. Voice direction of arrival and gaze tracking were used to determine when the Furhat robot should speak and when the participants were talking to each other. The way we handled the text generation and the robot’s speech gave some speed improvements and allowed participants to interrupt the robot’s speech if needed. First online 15 Jan 2025.