Conversational Agents: A Framework for Evaluation (CAFE)
Christine Bauer, Li Chen, Nicola Ferro, Norbert Fuhr, Avishek Anand, Timo Breuer, Guglielmo Faggioli, Ophir Frieder, Hideo Joho, Jussi Karlgren, Johannes Kiesel, Bart P. Knijnenburg, Aldo Lipani, Lien Michiels, Andrea Papenmeier, Maria Soledad Pera, Mark Sanderson, Scott Sanner, Benno Stein, Johanne R. Trippas, Karin Verspoor & Martijn C. Willemsen. 2025. “Conversational Agents: A Framework for Evaluation (CAFE): Manifesto from Dagstuhl Perspectives Workshop 24352”. In Dagstuhl Reports , Vol. 14, Issue 8, pp. 53–58. Schloss Dagstuhl – Leibniz-Zentrum für Informatik. DOI: 10.4230/DagRep.14.8.53
Abstract
This report documents the outcomes of Dagstuhl Perspectives Workshop 24352, titled “Conversational Agents: A Framework for Evaluation (CAFE)”, which brought together 22 researchers and practitioners from 12 countries.
From their discussions, the workshop developed a framework for evaluating conversational information access systems (CONIAC), centered on six core components: stakeholder goals, user tasks, user aspects, evaluation criteria, methodology, and measures.