Multi-party open-ended conversation with a social robot

TL;DR

In this work we explore wether an LLM can guide a multi-party spoken interaction. To track speakers, in our setting face recognition performed best; voice recognition was unreliable due to frequent short utterances and overlapping speech. The response and its addressee were generated by the LLM. Voice direction of arrival and gaze tracking were used to determine when the Furhat robot should speak and when the participants were talking to each other. The way we handled the text generation and the robot’s speech gave some speed improvements and allowed participants to interrupt the robot’s speech if needed. First online 15 Jan 2025.

What the system needs to do

Supporting open-ended, multi-party conversation leads to a few concrete requirements:

• Track the discourse and attribute every utterance to the right person;
• Know when to speak, and when users are talking to each other, when to stop speaking (interruptions), and when to resume;
• Use non-verbal cues like gaze and head movements to clarify turn shifts and addressee;
• Respond quickly;
• Stay modular and hardware-independent.

Modular architecture

We built the system as a set of modules that communicate through events. Results are published as soon as they are available; a central conversation manager combines them and decides what the robot says and to whom.

Speaker awareness

A ReSpeaker microphone array placed in front of the participants gives us the direction of arrival of each voice. This module emits events for detected speaker angles, turn changes, and the start and end of both user and robot speech.

Note: with the manufacturer’s casing on, the direction-of-arrival data was completely inaccurate. We removed the casing for the study.

Transcription

Speech is transcribed in real time with Azure Cognitive Services. The module records and transcribes each utterance continuously and keeps track of the last spoken word. Because it receives the speaker-awareness events, it knows when the robot itself is talking and avoids transcribing the robot’s own speech.

Diarisation and face tracking

Participants are identified through two channels:

• Voice: Azure’s speaker recognition identifies voices against profiles. Each participant goes through a short enrolment step before the conversation, reading one scripted sentence aloud.
• Face: Furhat’s built-in face tracking, combined with the Python face_recognition library, recognises participants. Faces are enrolled in a similar way.

The two are combined using the voice’s direction of arrival and the position of each face in the camera frame. When they disagree, the face wins, since face recognition proved to be the more reliable of the two.

Turn-taking

The turn-taking module uses face orientation and user position to decide when the robot should take the turn. The rule is simple: the robot speaks when the last speaker looks at it, or after a prolonged silence. This relies on the assumption that people pass the turn with their gaze. More sophisticated turn-taking models are an obvious next step.

Interruptions are handled explicitly. If a user starts talking while the robot has the turn, the robot pauses and yields. If the user stops again, the robot resumes where it left off.

Interactions

This module drives the robot. It uses Furhat’s text-to-speech for verbal output and controls gaze and head movements, looking towards whoever is currently talking. It is also where interruptions take effect: the robot’s speech pauses as soon as a user begins to talk. Furhat can also display facial expressions, but for this study we focused on gaze and head movements as the interaction tools.

Conversation manager

The conversation manager is the centre of the system. It uses an LLM (GPT-3.5, the latest version available when we ran the study in early 2024) to generate responses and keeps track of the conversation. It receives:

• the transcript of each utterance, tagged with its author on a best-effort basis;
• the list of people currently recognised by face tracking;
• the turn-taking signals.

From these, the LLM produces both a response and an explicit addressee. When speaker recognition fails for an utterance, the model infers the most likely speaker from context. In practice, this turned out to be one of the most important properties of the system.

To keep latency down, we use the LLM in streaming mode and send each sentence to speech synthesis as soon as it is complete, without waiting for the full response. When the robot is interrupted, only the part of the answer that was actually spoken is saved to the history, so the model’s view of the conversation matches what the users heard. The exact prompt is in the supplementary material.

We did not use incremental speech recognition: combined with the multi-party setting and the modular design, it added too much complexity for this first system. A natural next step is to start generating a response from partial transcripts and then use, extend or discard it once the robot actually takes the turn.

Evaluating it with 30 people

We recruited 30 participants in 15 pairs who already knew each other, so the conversations would feel natural. Each pair took part in two scenarios, in counterbalanced order, of about 12 minutes each:

• Parallel: the robot held two separate conversations, alternating between the two participants, each with their own topic. This is similar to a busy receptionist.
• Group: the robot and both participants had one shared, open discussion, with no rules about who speaks when.

We annotated the recordings for speaker identification, addressee accuracy and goal tracking, logged response latency, and collected questionnaires (RoSAS, Interaction Quality, Multidimensional Measure of Trust) and open comments.

Results

Mean response latency was 1.35 s (SD 0.55), excluding speech transcription time, sometimes close to 5 s. The LLM accounted for most of it, about 0.76 s on average.

The main insight are, first, that the LLM covers for perception gaps. Then, that the robot’s gaze and head movements must match what it says, and finally that we should focus on reducing latency and making the robot’s memory visible in the conversation.

Abstract

Multi-party open-ended conversation remains a major challenge in human–robot interaction, particularly when robots must recognise speakers, allocate turns, and respond coherently under overlapping or rapidly shifting dialogue. This paper presents a multi-party conversational system that combines multimodal perception (voice direction of arrival, speaker diarisation, face recognition) with a large language model for response generation. Implemented on the Furhat robot, the system was evaluated with 30 participants across two scenarios: (i) parallel, separate conversations and (ii) shared group discussion. Results show that the system maintains coherent and engaging conversations, achieving high addressee accuracy in parallel settings (92.6%) and strong face recognition reliability (80–94%). Participants reported clear social presence and positive engagement, although technical barriers such as audio-based speaker recognition errors and response latency affected the fluidity of group interactions. The results highlight both the promise and limitations of LLM-based multi-party interaction and outline concrete directions for improving multimodal cue integration and responsiveness in future social robots.

@ARTICLE{10.3389/frobt.2026.1766383,
AUTHOR={Abbo, Giulio Antonio  and Pinto-Bernal, Maria Jose  and Catrycke, Martijn  and Belpaeme, Tony },
TITLE={Multi-party open-ended conversation with a social robot },
JOURNAL={Frontiers in Robotics and AI},
VOLUME={Volume 13 - 2026},
YEAR={2026},
URL={https://www.frontiersin.org/journals/robotics-and-ai/articles/10.3389/frobt.2026.1766383},
DOI={10.3389/frobt.2026.1766383},
ISSN={2296-9144},
}