Title: User studies of knowledge acquisition tools: methodology and lessons learned
Authors: Tallis, Marcelo., Kim, Jihie. & Gil, Yolanda
Subject: Knowledge management
Publish: 2001
Status: full text
Source: Journal of Experimental & Theoretical Artificial Intelligence; Oct-Dec 2001, Vol. 13 Issue 4, p359-378
Preparation: Scientific Database Management Journal Articles www.SYSTEM.parsiblog.com
Abstract: Knowledge acquisition research concerned with the development of knowledge acquisition tools is in need of a methodological approach to evaluation. This paper describes experimental methodology to conduct studies and experiments of users modifying knowledge bases with knowledge acquisition tools. The paper also reports on the lessons learned from several experiments that have been performed using this methodology. The hope is that it will help others design user evaluations of knowledge acquisition tools. Ideas are discussed for improving the current methodology and some open issues that remain.   --Download Article
Keywords: knowledge management; knowledge acquisition.

Introduction: The field of artificial intelligence has increasingly recognized throughout the years the need and the value of being an experimental science. Some subfields have developed standard tasks and test sets that are used routinely by researchers to show new results. Researchers in machine learning, for example, use the Irvine data sets (Blake and Merz 1998) to show improvements in inductive learning (Quinlan 1993, Webb et al. 1999) and routinely use tasks like the n-puzzle or the n-queens for speeding-up learning research (Tambe and Rosenbloom 1990, Kim and Rosenbloom 1996).
Developing standard tests is harder in other subfields that address more knowledge-intensive problems. For example, planning researchers often show experiments in similar task domains (Gil 1991, Gil 1992, PeArez and Carbonell 1994, Estlin 1998).
The problem is that the implementation of the knowledge base (KB) and of the algorithms is so di? erent across systems that the results of the experiments are often hard to analyze. One approach used by some researchers is to use artificially created, very structured knowledge bases to analyze particular behaviors . Another approach has been to define a universal language to describe planning domains, as is done in the Planning Competition of the Artificial Intelligence Planning Systems Conference (McDermott 2000).
Knowledge acquisition (KA) research has a traditional focus on even more knowledge-intensive problems. Di? erent systems use a wide variety of representations and are often built to address di? erent aspects of KB reasoning as well as to acquire di? erent kinds of knowledge. In recognition of the need to evaluate KA research, the community started to design a set of standard task domains that di? erent groups would implement and use to compare their work. This e? ort is known as the Sisyphus experiments (Linster 1994, Schreiber and Birmingham 1996, Shadbolt et al. 1999), and the domains have included o? ce assignment, elevator configuration and rock classification. These experiences have been useful to illustrate particular approaches, but have not served in practice as testbeds for comparing and evaluating di? erent approaches (Gil and Linster 1995).
As developers of knowledge acquisition tools we wanted to evaluate our approach, and began looking into user studies. With the exception of some isolated evaluations of KA tools (Joseph 1992, Yost 1992, Murray 1995), we found that the field of knowledge acquisition had no methodology that we could draw from to design our evaluations. Even though artificial intelligence is, as we mentioned earlier, a field where experimental studies have been increasingly emphasized in recent years, user studies are uncommon. User studies to evaluate software tools and interfaces can be found in the literature of tutoring systems (Self 1993), programming environments (Basili et al. 1986) and human computer interfaces (Olson and Moran 1998). These communities are still working on developing evaluation methodologies that address their specific concerns. All seem to agree on the di? culty and cost of these studies, as well as on their important benefits. Often the evaluations that test specific claims about a tool or approach are not as thorough or conclusive as we would like to see as scientists, yet these evaluations are very valuable and are shedding some light on topics of interest (Rombach et al. 1992, Self 1993, Basili et al. 1986, Olson and Moran 1998). In developing a methodology for evaluation of KA tools, we can draw from the experiences that are ongoing in these areas.
The lack of evaluation in knowledge acquisition research is unfortunate, but could be due to a variety of reasons. First, user evaluations are very costly. In areas like machine learning and planning, experiments often amount to running programs repeatedly on already existing test sets. The evaluation of a KA tool requires that a number of subjects spend a fair amount of time doing the study, and for the experimenters to spend time and other resources preparing the experiment (often months) and analysing the results. The most recent Sisyphus is an example of the issue discussed above about the intimidating cost of KA evaluations: the limited number of participants can be tracked back to the significant amount of resources required to tackle the knowledge-intensive task that was selected (Shadbolt et al. 1999). Second, most of the research in the field of KA concentrates on knowledge modelling (e.g. how a knowledge engineer models a task domain) and knowledge elicitation (e.g. techniques for interviewing experts). There are very few e? orts on developing tools for users. KA tool developers may have conducted usability studies, but the results are not reported in the literature. Third, unless human experiments are carefully designed and conducted, it is hard to draw conclusive results from the data.
Over the last few years, we have performed a series of user evaluations with our KA tools that have yielded not only specific findings about our tools but that have also allowed us to develop a methodology that we follow in conducting evaluations (Gil and Tallis 1997, Kim and Gil 1999, Tallis 2000, Tallis and Gil 1999, Kim and Gil 2000a, Kim and Gil 2000b). This paper describes our experimental methodology to conduct studies of users modifying knowledge bases with KA tools. It also reports the lessons learned from our experiments, so it will help others design or improve future user evaluations. This paper describes our experiments in enough detail to illustrate the main points of our methodology. A more comprehensive deion of our experiments and their results can be found in the above references. The paper describes our experiences based on tests with two particular KA tools
that we developed for EXPECT (Gil 1994, Swartout and Gil 1995, Gil and Melz 1996). EXPECT is a framework for developing and modifying knowledge-based\ systems (KBSs), whose main purpose is to enable domain experts lacking computer science or artificial intelligence background to directly manipulate the elements of a KB without the mediation of a knowledge engineering. The two tools that were the subject of our evaluation were intended to enhance some aspect of the EXPECT support to end users. ETM (EXPECT transaction manager) (Gil and Tallis 1997, Tallis 2000, Tallis and Gil 1999) uses typical KB modification sequences (KA s) to help users make complex changes that require many steps to modify KBs. EMeD (EXPECT method developer) (Kim and Gil 1999, Kim and Gil 2000a, Kim and Gil 2000b) analyses and exploits interdependencies among KB elements to create expectations about how new knowledge fits and detect missing knowledge that  eeds to be acquired from users. Each tool was developed to investigate a di? erent approach to guide users in knowledge acquisition tasks. The approaches are complementary, and we have recently integrated the features of the tools that we found useful in the experiments in order to create a more comprehensive and powerful KA environment for EXPECT (Blythe et al. 2001). A brief overview of both tools can be found in Appendix A. Please note that the focus of this paper is not on the details of these tools but on our experimental methodology and the lessons learned from our experiments.
The paper begins by describing the methodology that we follow to evaluate KA tools, illustrated with examples from our evaluations with ETM and EMeD. The next section highlights the lessons that we learned in carrying out our initial experiments, and describes open issues in KA experiment design. Finally, we discuss related work in KA and in other research disciplines that conduct user studies, and outline directions for future work.

2. A methodology to conduct experimental user studies with knowledge acquisition
tools
The nature of an experiment is determined by the questions that it will help answer or understand. In our case, these are stated as claims and hypotheses about our tools. The hypotheses to be tested determine what are the KA tasks to be performed by users, the type of users involved, the procedure to be followed to perform the experiment, and the data that needs to be collected. This section discusses each of these issues in detail.
Table 1 summarizes the steps in our methodology. It is by no means a strictly sequential process, rather there is significant iteration and backtracking across these steps due to the interactions among all the constraints and decisions involved. For example, a hypothesis may be revisited if an experiment cannot be designed to test it as it is stated.
2.1. Stating claims and hypotheses
Claims and hypotheses play a pivotal role in the evaluation process, since the experiments revolve around them. Claims and hypotheses are related but not necessarily the same. Claims are stated in broader terms, referring to general capabilities and benefits of our tools. It is often not possible to test a broad claim, but formulating it helps us understand what we think are the advantages of a certain approach. Based on these broader claims, we formulate specific hypotheses that we would like to test. In contrast with claims, hypotheses are stated in specific terms, and we formulate them so that an experiment can be designed to test them and yield evidence that will lead to proving or disproving specific hypotheses.
Note that many experiments are designed to explore how something works, without any specific hypotheses or claims. For example, several alternative interface designs can be evaluated with users to find out which designs are more suitable, perhaps without any prior hypothesis about which features are best. The first step in the design of our evaluations is to state the main claims regarding our KA tools. We ended up formulating similar claims for both ETM and EMeD:
 (1) Users will be able to complete KA tasks in less time using the KA tools. Rationale: our KA tools would support some time consuming activities involved in KB modification tasks. For example, they support the analysis of the interactions among the elements of KB and the choice of actions to remove inconsistencies in the KB.
 (2) Users will make less mistakes during a KA task using the KA tools. Rationale: our KA tools detect missing knowledge and inconsistencies in the KB and they also support users in fixing them.
 (3) The reduction in completion time and number of mistakes will be more
noticeable for less experienced users. Rationale: less experienced users will be the most benefited from the tool’s thorough guidance in making changes to a KB. The tools will also be able to resolve the inconsistencies that arise during the modification of the KB using
strategies that may be unknown to less experienced users but are well-known to more experienced ones.
(4) The reduction in time will also be more noticeable for users lacking a detailed knowledge of the KBS implementation. Rationale: our tools detect interactions with already existing knowledge. Our tools also reveal the existence of KB elements that can be reused or adapted
and that the users may not be aware of. This should be particularly noticeable when the KB is large.
(5) The KA tools will be useful for a broad range of domains and knowledge
acquisition scenarios. ationale: our tools are based in general domain-independent principles. Given these claims, we were able to state specific and measurable hypotheses to be proved or disproved with experiments that were feasible given our resources and constraints. For example, a specific hypothesis for ETM corresponding to claim 1 is: completion time for a complex KBS modification will be shorter for subject using ETM in combination with the EXPECT basic KA tool than for subjects using the EXPECT basic KA tool alone.
A claim can be stated in more general or more specific terms depending on the purpose of the claim. The claims shown above are specific to particular KA tools and methodologies, but it would be useful to make them part of more general claims that the whole KA field cares about and that other researchers may want to hear about the state-of-the-ar t in KA. For example, our experiments and those of others might help us gather evidence towards general claims such as `It is possible for naive users to make additions and changes to a knowledge base using current state-of-the-art KA technology’, with more specific claims stating what technologies help in what kinds of KA tasks to what kinds of users.
2.2. Determining the set of experiments to be carried out
It is useful to test one or more hypotheses in a few experiments, but it is not always possible. This is the case when the hypotheses are of a very di? erent nature, or when a given hypothesis needs to be tested over a range of user types, tasks or knowledge bases. For example, if we had two di? erent hypotheses, such as (1) a KA tool helps to perform a task more e? ciently and (2) the KA tool scales up to large and realistic applications, then it might be necessary to conduct one experiment for the first hypothesis and a very di? erent experiment for the second hypothesis. In practice, many hypotheses are hard to evaluate because they imply experiments that may be unfeasible due to lack of time and other resources. In order to show the benefits of a tool or technology, a useful way to design an experiment is to perform a comparison with some baseline tool. In this case, we have to carefully choose the baseline tool so that the only di? erence between the two tools is the presence or absence of the technology to be evaluated. Otherwise, we may not be able to determine if the di? erences in performance were due to the technology itself or to some other factors (e.g. a di? erent interface design or interaction style). Comparing the performance of users using a KA tool with users using an editor to enter knowledge is only useful if the hypothesis is that using a KA tool is better than not using it at all, which is normally not a hypothesis that one questions in an experiment…
--Download Article