Handbook for Evaluating Knowledge-Based Systems Conceptual Framework and Compendium of Methods

Knowledge-based systems are increasingly found in a wide variety of settings and this handbook has been written to meet a specific need in their widening use. While there have been many successful applications of knowledge-based systems, some applications have failed because they never received the corrective feedback that evaluation provides for keeping development focused on the users' needs in their actual working environment. This handbook provides a conceptual framework and compendium of methods for performing evaluations of knowledge-based systems during their development. Its focus is on the users' and subject matter experts' evaluation of the usefulness of the system, and not on the developers' testing of the adequacy of the programming code. The handbook permits evaluators to systematically answer the following kinds of questions: Does the knowledge-based system meet the users' task requirements? Is the system easy to use? Is the knowledge base logically consistent? Does it meet the required level of expertise? Does the system improve performance?
The authors have produced a handbook that will serve two audiences: a tool that can be used to create knowledge-based systems (practitioners, developers, and evaluators) and a framework that will stimulate more research in the area (academic researchers and students). To accomplish this, the handbook is built around a conceptual framework that integrates the different types of evaluations into the system of development process. The kinds of questions that can be answered, and the methods available for answering them, will change throughout the system development life cycle. And throughout this process, one needs to know what can be done, and what can't. It is this dichotomy that addresses needs in both the practitioner and academic research audiences.

1 Introduction.- 1.1 Defining Terms.- 1.2 The Role of Evaluation.- 1.3 Purpose of Book.- 2 Overview: Development Models, Evaluation Dimensions, and Evaluation Models.- 2.1 Life Cycle Development Models.- 2.2 Evaluation Dimensions, Criteria, and Attributes.- 2.3 Knowledge-Based System Evaluation Models.- 2.4 Chapter Summary.- 3 Framework for Evaluating Knowledge-Based Systems.- 3.1 Evaluation Issues.- 3.2 Generic Development Stages.- 3.3 Sources of Evaluation Data.- 3.4 Types of Evaluations.- 3.5 Evaluation Constraints.- 3.6 Chapter Summary.- 4 Requirements Validation.- 4.1 Requirements Validation Framework and Issues.- 4.2 Requirements Validation Methods.- 4.3 Method Applicability.- 4.4 Summary.- 5 Knowledge Base Validation.- 5.1 Conceptual Issues and Terms.- 5.2 Knowledge Base Validation Methods.- 5.3 When to Examine the Knowledge Base and Use Test Cases.- 5.4 Chapter Summary.- 6 Knowledge Base Verification.- 6.1 Operationalizing KB Verification.- 6.2 Verification Methods.- 6.3 Knowledge Base Management.- 6.4 Supporting Tools.- 6.5 Organizational Use.- 6.6 Recommendations.- 6.7 Conclusions: Good Versus Bad Practice.- 7 Usability Evaluation.- 7.1 Usability Engineering and Evaluation: Two Perspectives.- 7.2 Expert Evaluation Methods.- 7.3 Subjective Evaluation Methods.- 7.4 Empirical Evaluation Methods.- 7.5 Relative Effectiveness of Usability Evaluation Methods.- 7.6 Chapter Summary.- 8 Performance Evaluation.- 8.1 Conceptual Issues.- 8.2 Performance Evaluation Methods.- 8.3 When to Use Different Performance Evaluation Methods.- 9 Planning and Managing Knowledge-Based System Evaluations.- 9.1 Steps in Planning and Managing the Evaluations.- 9.2 Implementing Life Cycle Evaluations: Problems and Solutions.- 9.3 Recommendations and Conclusion.- References.- Author Index.