Preprint
Speech & Audio

Towards a tool for the Subjective Assessment of Speech System Interfaces (SASSI)

Kate Hone(Brunel University of London), Robert Graham(Motorola (United Kingdom))
September 1, 2000Natural Language Engineering346 citations
PDFDOI

346

Citations

32

Influential Citations

Natural Language Engineering

Venue

2000

Year

Abstract

Applications of speech recognition are now widespread, but user-centred evaluation methods are necessary to ensure their success. Objective evaluation techniques are fairly well established, but previous subjective techniques have been unstructured and unproven. This paper reports on the first stage in the development of a questionnaire measure for the Subjective Assessment of Speech System Interfaces (SASSI). The aim of the research programme is to produce a valid, reliable and sensitive measure of users' subjective experiences with speech recognition systems. Such a technique could make an important contribution to theory and practice in the design and evaluation of speech recognition systems according to best human factors practice. A prototype questionnaire was designed, based on established measures for evaluating the usability of other kinds of user interface, and on a review of the research literature into speech system design. This consisted of 50 statements with which respondents rated their level of agreement. The questionnaire was given to users of four different speech applications, and Exploratory Factor Analysis of 214 completed questionnaires was conducted. This suggested the presence of six main factors in users' perceptions of speech systems: System Response Accuracy, Likeability, Cognitive Demand, Annoyance, Habitability and Speed. The six factors have face validity, and a reasonable level of statistical reliability. The findings form a useful theoretical and practical basis for the subjective evaluation of any speech recognition interface. However, further work is recommended, to establish the validity and sensitivity of the approach, before a final tool can be produced which warrants general use.

Analysis

Why This Paper Matters

This paper addresses a critical gap in the evaluation of speech recognition systems: the lack of a validated, structured subjective assessment tool. While objective metrics like word error rate are well-established, they fail to capture the user's holistic experience, which is essential for system adoption and satisfaction. The authors systematically develop and test a questionnaire (SASSI) that aims to measure subjective perceptions reliably. Given the widespread deployment of speech interfaces in the late 1990s and early 2000s, this work provided a much-needed human factors perspective, moving beyond technical accuracy to usability and user experience.

The significance lies in its rigorous approach to questionnaire development, borrowing from established usability measures for other interfaces and grounding items in speech-specific literature. By identifying six distinct factors (Accuracy, Likeability, Cognitive Demand, Annoyance, Habitability, Speed), the paper offers a multidimensional view of user experience, which is more informative than a single satisfaction score. This framework has influenced subsequent research and practice in speech interface evaluation.

Technical Contributions

  • Structured questionnaire design: The 50-item prototype was built on established usability measures and a thorough literature review, ensuring content validity.
  • Factor identification: Using Exploratory Factor Analysis on 214 responses, the study extracted six interpretable factors that capture key dimensions of user perception.
  • Statistical reliability: The factors demonstrated reasonable internal consistency (Cronbach's alpha), supporting the reliability of the proposed structure.
  • Practical framework: The six factors provide a clear, actionable set of dimensions for evaluating and comparing speech systems from a user perspective.

Results

The factor analysis yielded six factors with face validity and reasonable statistical reliability. The specific factor loadings and reliability coefficients are not detailed in the abstract, but the overall structure was deemed sound. The study involved users of four different speech applications, lending some generalizability. However, the abstract notes that further validation and sensitivity testing are required before the tool is ready for general use.

Significance

This paper laid the groundwork for a standardized subjective evaluation tool in speech recognition, influencing both academic research and industry practice. It highlighted the importance of user-centered design in speech interfaces and provided a replicable methodology for developing similar instruments. The SASSI questionnaire has been cited in numerous subsequent studies and remains a reference point for subjective speech system evaluation. Its multidimensional approach encourages designers to consider not just accuracy but also cognitive load, annoyance, and other experiential factors.

speech-recognitionuser-interfacequestionnaireusabilitysubjective-assessmenthuman-factorsevaluation

Related Papers