# Assessing LLMs Responses in the Field of Domestic Sustainability: An Exploratory Study

- Authors: Mathyas Giudici, Giulio Antonio Abbo, Ottavia Belotti, Alessio Braccini, Francesco Dubini, Riccardo Andrea Izzo, Pietro Crovari and Franca Garzotto
- Date: 2023-11-27
- Type: conference paper
- Venue: Third International Conference on Digital Data Processing (DDP '23)
- DOI: http://dx.doi.org/10.1109/DDP60485.2023.00019
- Code: ddd
- Supplementary material: https://doi.org/10.5281/zenodo.10127383
- ResearchGate: https://www.researchgate.net/publication/378158929_Assessing_LLMs_Responses_in_the_Field_of_Domestic_Sustainability_An_Exploratory_Study
- PDF: https://giubots.net/publications/assessing-llms-responses/assessing-llms-responses.pdf
- Research line: [Collabs](https://giubots.net/research/collabs/)
- URL: https://giubots.net/publications/assessing-llms-responses/


## TL;DR

This study evaluates how well large language models can answer questions about domestic sustainability. ChatGPT was found to be the most suitable for integrating into smart home assistants due to its fluency and accessibility, though experts noted its answers sometimes lacked factual accuracy.


This study evaluates how well large language models can answer questions about domestic sustainability. ChatGPT was found to be the most suitable for integrating into smart home assistants due to its fluency and accessibility, though experts noted its answers sometimes lacked factual accuracy.

## Abstract

In the next years, we must challenge climate change, and the urgency of adopting a more sustainable lifestyle has increased. Conversational Agents, such as Smart home Personal Assistants, have shown promise in fostering sustainable behaviors in domestic environments. However, traditional conversations with rule-based approaches in such agents face challenges in addressing users' questions in complex domains like sustainability. Large Language Models (LLMs) are a promising tool to overcome these limitations of their capability to answer open-domain questions. The final objective of this work is to compare the generative capabilities of four large language models in ecological sustainability to determine the most suitable LLM to be embedded into home assistants and create a hybrid model of conversational agent for environmental sustainability. We performed two evaluations. In the former, we constructed a set of trustable sources on the topic and analyzed the extent to which the themes covered in the text generated by the models appeared in it. The results do not show a statistical difference between the outputs of the candidate models, while qualitative analysis determined that ChatGPT, at the moment, is the optimal solution. In the second evaluation, we tested the responses generated by ChatGPT on a corpus of 167 questions from a sample of 75 people. Responses evaluation was performed by a team of experts (N=5) on fluency, coherency, consistency, accuracy, and reasoning. The results suggest that ChatGPT for generic questions on sustainability is quite reliable.

## How to cite

Mathyas Giudici, Giulio Antonio Abbo, Ottavia Belotti, Alessio Braccini, Francesco Dubini, Riccardo Andrea Izzo, Pietro Crovari, and Franca Garzotto. 2023. Assessing LLMs Responses in the Field of Domestic Sustainability: An Exploratory Study. In 2023 Third International Conference on Digital Data Processing (DDP), 42–48. DOI:https://doi.org/10.1109/DDP60485.2023.00019

```bibtex
@INPROCEEDINGS{giudici2023assessing-llms-responses,
author={Giudici, Mathyas and Abbo, Giulio Antonio and Belotti, Ottavia and Braccini, Alessio and Dubini, Francesco and Izzo, Riccardo Andrea and Crovari, Pietro and Garzotto, Franca},
booktitle={2023 Third International Conference on Digital Data Processing (DDP)}, 
title={Assessing LLMs Responses in the Field of Domestic Sustainability: An Exploratory Study}, 
year={2023},
volume={},
number={},
pages={42-48},
keywords={Climate change;Sustainable development;Biological system modeling;Chatbots;Analytical models;Smart homes;Environmental monitoring;Modeling;Oral communication;Agent-based modeling;Statistical analysis;Statistical distributions;Trust management;Conversational Agent;Sustainability;LLM;Rule-based CA},
doi={10.1109/DDP60485.2023.00019}}
```