[ Abstract | Introduction | Methods | Results & Discussion | Conclusion | References | Bibliography ]


The Knowledge Integration Environment (KIE) is a computer-based learning environment combining online tools such as a multimedia discussion forum (SpeakEasy) and on-demand cognitive and procedural guidance. KIE encourages students to develop an expanding repertoire of alternatives that are evaluated through the application of principled knowledge. The KIE environment is based on the Scaffolded Knowledge Integration (SKI) framework (Linn, 1995) that specifies four goals for learning environments: (a) identify new goals for learning, (b) make thinking visible, (c) encourage autonous learning, and (d) provide social supports. These goals are addressed in the design/search activities by encouraging iterative refinement of designs, by helping students locate Internet sites independently, and by providing tools for sharing Internet resources. Through a process of trial and refinement, the studies discussed in this paper extend research in the areas of collaborative design, pragmatics and communication, category and concept representation, environmental science education, and information retrieval.

In the first experiment (hereafter referred to as the Relevance Judgment Experiment), eighth graders (N=139) searched the World Wide Web (WWW) for information relevant to a design task involving the storage of heat and the regulation of temperature. The activity consisted of surveying selected Internet sites related to energy conservation and environmentally-sensitive housing design, searching the Internet for evidence to support original designs, and formulating a final report integrating drawings of a desert house with principled evidence for design decisions. The majority of the students were successful at searching, locating, and incorporating Internet evidence into the design process (e.g., 67% of the groups located useful sites). Students varied in sites they found useful; those sites rated as the most useful were rated as not useful by other students. This inconsistency in ratings for sites motivated the second study that looks at how to improve the search process through collaboration and shared access to information.

In the second experiment (hereafter referred to as the Collaborative Search Experiment), 140 eighth graders performed the design task. A Collaborative Search Page (CSP) was added where sites selected by students were automatically added to a publicly-accessible and searchable Web form. Log files were generated to track student activity patterns, queries, and comments about Internet sites. The new activity design was successful in expanding the students' definition of the problem as measured by the number of groups that located sites they thought were useful (e.g., 83% an increase of 16%). The publicly-accessible and searchable Web form helped scaffold those students that had difficulty locating Internet material on their own. Extending earlier work in the area of information resources and design tasks (Cuthbert, 1996), this research confirms the hypothesis that the mechanisms that help students critically analyze marginally relevant data and develop cogent, principled arguments are tied to the ability to conceptualize the problem as opposed to the ability to generate queries to narrow the search space.

Design Framework.The interesting question which this paper addresses then becomes: Do students discard information because they have a narrow conceptualization of the problem or because they review evidence superficially? To answer this question I begin to develop a model of how poorly-structured problems are solved and specifically how information resources are utilized during the design process. Ultimately these models will be used to present users with customized learning environments based on a statistically computed similarity matrix that matches user profiles. This paper examines the potential for segmenting user profiles based on a categorical decomposition of interactional dispositions[1] (IDs). The premise is that these ID groups may provide a starting point for routing users along different paths in the learning environment.

Student-Centered Perspectives. Collaborative filtering has emerged as the prevalent term for the evaluation of information resources by communties of practitioners. In my work, the Collaborative Search Page (CSP) serves as a repository for Internet sites that have been located and evaluated by members of the student community working on a common design project. The CSP combines ideas from spreading activation theories[2] (Pirolli , 1996; Card, 1996) that suggest that similar users interact with a small set of information resources.

My research extends work in hypermedia navigation and search models in two significant areas. First, by using the Internet, an unconstrained and poorly-indexed network of information rather than a relational database, the extent of the available resources is not clearly-bounded. Second, I focus on the process of teams moving through both local and remote sites rather than patterns within a specific locality (Pirolli, 1996). Modeling end-user activity provides a perspective on the activity patterns observed within a specific locality that is not possible when looking solely at traffic within a single Web site. In effect, this research attempts to provide explanations for the activity patterns in addition to identifying them.

Locating Evidence.To help students locate evidence, a combination of teacher-guided keyword generation and "likely result" analysis precedes the actual search by individual groups. The software environment provides a mechanism (e.g., the Collaborative Search Page) to share relevant resources with the goal of reducing the imprecision and low recall rates of searches. When an Internet site is saved, students enter metadata such as the perceived usefulness and reliability of the sitein a dialog box entry form. Pooling relevance judgments (Landauer, 1991) solves the problem where there is a lack of congruence between terms assigned by users to represent the same item[3]. Other methods for solving this problem include adaptive indexes (Furnas, 1985), latent-semantic indexing (Furnas et. al., 1983, 1987), and context-guided search (Landauer, 1991). Underlying all of these studies is the premise that how to share relevent information is the real problem with searching rather than finding ways to develop faster or better search algorithms. The question then becomes how to define and connect communities of practise.

Using Evidence. On a pedagogical level, this research evaluates various mechanisms that help students use information resources for problem solving, theory comparison, and design tasks. These mechanisms include tools that help students critique Internet sites along with activity structures that encourage the formulation of well-structured arguments through iterative refinement. Effective performance requires: (a) elaboration of the problem and (b) linking problem components with principles. Students that do not use evidence to generate alternatives should have limited success in searching for useful information and ultimately in developing a well-structured problem solution. This argument is falsifiable if students with constrained problem definitions can locate and use evidence effectively.

The Nature Of Design Problems. Design problems differ from constrained problem-solving situations in several ways. First, design problems do not have logical rules of transformation or a fixed set of operators. Second, there is no single correct solution but multiple solutions that have varying degrees of resolution depending on the constraints that are formulated. Finally and perhaps most significantly, in design tasks the problem is often under-specified. The ill-structured nature of design tasks means that a problem structuring phase preceeds the preliminary design, refinement, and detail design phases (Goel, 1996).

Previous research suggests that effective searching is tied to the problem definition (Cuthbert, 1996). Most students neglect problem structuring and move directly to generating design decisions. Evidence evaluation conducted prior to making design decisions should be more exploratory in style. Frequently, information in the preliminary design phase is evaluated in the process of generating options. A search activity conducted after the preliminary design phase, as occurred in these experiments, is typically conducted to locate evidence to support an already existing design. However, for certain types of learners post hoc searches can result in substantial revision of existing designs.

Four possible actions are postulated for how students deal with conflicting or alternative ideas: refinement, replacement, disregard, or aggregation. Because I expect a priori searches to be more exploratory in style, students should be less likely to use that evidence to justify their design decisions. The reverse should be true for post hoc searches. This research ibegins to explore how these approaches vary based on the phase of the design process.

The points at which options are generated vs. those where designs coalesce appear to vary based on a variety of factors. Since the conceptualization of the problem is often based on pieces of evidence encountered early in the design process aggregated with prototypical or intuitive representations of the object to be designed, particular attention must be paid to the sequence in which students encounter information. I expect the interactional disposition of the student to greatly influence the persistence of the initial conception and the likelihood that additional information will alter or replace that initial formulation.

Linn's (in press) four high-level categories of interactional dispositions (e.g., conceptualizer, experimenter, strategizer, and consumer) provide useful divisions for analyzing and predicting the effect of information resources on designs. Conceptualizers typically apply principles or rules-of-thumb when solving problems while experimenters maintain a repertoire of alternatives that they explore. Strategizers seek out what they think will be the easiest and most acceptable solution - a type of cost-benefit analysis. Consumers adopt solutions without critically analyzing the alternatives either by completely adopting the new alternative or adding elements to the existing design without justification. The planned analysis based on interactional dispositions attempts to draw parallels between evidence usage in the final reports and the patterns of activity during the search episode. Future research[4] will apply these findings by developing controlled experiments that test the effectiveness of routing users through the information landscape based on a number of parameters including their interactional dispositions.

[ Abstract | Introduction | Methods | Results & Discussion | Conclusion | References | Bibliography ]