[ Abstract | Introduction | Methods | Results & Discussion | Conclusion | References | Bibliography ]

Methodology

Both experiments discussed here involve the end-of-semester KIE "Desert Houses" activity that follows a series of computer-based labs on heat flow, insulation, and conduction. The subjects' task was to design an energy-efficient house for the desert that did not rely on external energy sources besides the sun. The house needed to remain cool during the day and warm at night. During the activity, the participants examined actual Internet sites, KIE-prepared sites[1], and excerpts from classroom research completed earlier in the semester (e.g., the Heat Bars lab that measured the conductivity of various materials). Students searched for relevant Internet sites by generating keywords as a class and then entering words into a form that fed into a search engine (i.e., Excite[2] or Lycos[3]).

The Relevance Judgment experiment looks at ways of expanding the problem definition with the goal of helping students locate evidence and incorporate that material into the final design reports. The Collaborative Search experiment examines the utility of a publicly-accessible Web page containing student-located Internet sites. Clinical interviews with eight students provide an in-depth look at the conceptual understanding behind the designs.

Participants. The eighth graders involved in this study had taken a semester of science using the Computer As A Learning Partner (CLP) curriculum.

Materials. The participants worked on Macintosh LC II computers using Netscape, Microsoft Works, and the KIE environment. The KIE environment links the various software components through a floating tools palette. An activity checklist and on-demand help facility provide procedural and cognitive guidance for students. A series of worksheets accompany the different phases of the activity and are integrated with the software components and online guidance.

Technical Configuration. The Macintosh LC II computers are on a local area network (LAN) connected to one classroom-based Macintosh file server along with a separate Web server. The file server contains the student projects while the Web server handles the forms management and CGI scripts (Common Gateway Interface). A direct connection to a DEC Alpha OSF/1 Unix server at Berkeley provides high-speed access to the Networked Evidence Database (NED) containing the advance organizers for the selected Internet sites that students access in the "Survey" phase of the activity.


Relevance Judgment Experiment


Procedure. An overview of the phases of the 6-day project and timetable follows:
Checklist                 Activity                 
Time                       
Survey                     Survey Three Houses        1 
day                      
Synthesize                 Synthesize Evidence        1 
day                      
Discuss                    SpeakEasy                  1.5 
days                   
Search                     Search Activity            1 
day                      
Design                     Design Report              1.5 
days                   

Figure 1. Desert Houses Activity Overview (First Experiment)

The projects were graded as a unit with a total of 80 possible points (A: 72-80; B: 64-71; C: 56-63.)

Formation Of Groups. Students were allowed to select their partners for the activity and worked in pairs or groups of three throughout the project.

Instructions To Participants. The Survey, Synthesize, and Online Discussion involved selected Internet sites related to the energy efficiency of wood, straw, and mud as building materials for houses.

At the beginning of the Search phase, the instructor asked the class to collectively generate and think about the types of words that could be used to locate Internet sites relevant to the project. Words such as "house" were discouraged because they were either too vague or would bring up irrelevant sites such as real estate companies. The instructor encouraged students to use more scientific terms such as "energy conservation" or "insulation." The Search Worksheet allowed students to track queries off-line by recording the keywords, usefulness ratings, evidence that was kept, and the reason for keeping it.


Key Words How good? Evidence we kept Why we kept it ________________ ___Very Useful Title: ___Building material ___Search Engine ___Somewhat Useful ___Desert House ___Thesaurus ___Not at all Useful ___Other

Figure 2. Search Worksheet

The online Search Form instructed students to find two pieces of evidence that were either: (a) examples of desert dwellings or (b) information about a building material that would work well in the desert. When students encountered sites they thought were useful, they pressed the "Save to Netbook" button. The on-line thesaurus provided lists of synonyms for keywords.

Experimental Manipulation. One of the sections (period 1) received slightly different instructions during the search phase and completed a preliminary design at the beginning of the activity. Rather than being instructed to find examples of desert dwellings or information about building materials, the assignment was to find at least 2 pieces of evidence that "could be used to support your reasons for designing a desert house one way instead of another. To start, think about what makes a house warm or cold in the desert. Remember that in the desert it is usually warm in the day and cool at night. On your worksheet list: five keywords that describe what makes a house warm or cold in the desert." The experimental group received a blank "Why We Kept It" column on the search worksheet as opposed to the categories "Building Material", "Desert House", and "Other".

Planned Analysis. A comparison of the frequency of keywords across groups with a weighted ranking of usefulness ratings will provide an indication of the uniformity with which students evaluate sites. The percentage of groups that saved sites that they thought were useful can be used as a metric for the success of locating information during the search activity. To measure the success of using that evidence, a breakdown of the information cited in the final reports (e.g., NED vs. Other/Search vs. Any) can be used to evaluate the hypothesis that information resources encountered early in the activity are used to generate alternatives vs. those found later in the activity that are used to support existing designs. Tracing the stimulus that resulted in innovative designs can provide insights into how students refine or substantially revise their designs.



Collaborative Search Experiment

Procedure. An overview of the phases of the 10-day project, timetable, and scoring follows:

Checklist Activity Time Points Sketch1* Preliminary Sketch .5 days 10 Survey* Survey Three 1 day 10 Houses Synthesize Synthesize 1 day 20 Evidence Discuss SpeakEasy 1.5 days 10 Quiz Synthesize 1 day 10 SpeakEasy Search* Search Activity 2 days 20 Sketch2* Heat Flow Analysis 1 day 30 Sketch3* Redesign .5 days 0 Design* Design Report 1.5 days 50

Figure 3. Desert Houses Activity Overview (Second Experiment)
(*phases analyzed in this paper)

There were a total of 150 possible points. The Sketches, Survey, Search, and Design phases were graded separately from the Synthesize, Discuss, and Quiz phases.

Formation Of Groups. Students were allowed to select their partners for the activity and worked in pairs or groups of three throughout the project.

Instructions To Participants. In the Preliminary Sketch phase, students listed three things that they thought were important in designing a dwelling for the desert that would stay cool in the day and warm during the night. In the preliminary sketch of what an energy efficient house might look like, the students used arrows to show heat flow during the DAY vs. the NIGHT.

The Survey, Synthesize, Online Discussion, and Quiz involved an expanded list of selected Internet sites some of which were located by students the previous semester (e.g., Specific Heat and Heat Capacity: Enertia Systems.)


Survey Evidence

Survey the evidence below and think about how it will influence your design of a house for the desert. You will be discussing which of the houses described below would keep people at a comfortable temperature, so read carefully.

MUST SEE

Science References

Example Houses

OPTIONAL

More Science References

More Example Houses



Figure 4. Survey Evidence Headings

At the beginning of the Search phase, the instructor asked the class to collectively generate and think about the types of words that could be used to locate Internet sites relevant to the project. Words such as "house" were discouraged because they were either too vague or would bring up irrelevant sites such as real estate companies. The instructor encouraged students to use more scientific terms such as "energy conservation" or "insulation". The Search Worksheet allowed students to track queries off-line by recording the keywords, usefulness ratings, evidence that was kept, and the reason for keeping it. The "Why We Kept It" column in Figure 1. was left blank as opposed to the checklist in the first study.

The online Search Form instructed students to find two pieces of evidence that were either: (a) examples of desert dwellings or (b) information about a building material that would work well in the desert. When students encountered sites they thought were useful, they pressed the "Save to Netbook" button. For this second study, they completed a site submission dialog that paralleled the Search Worksheet fields. The checkbox options for categories describing a site began with "Windows", "Color", and "Insulation" (the three topics used in the Survey Evidence section.) If students entered a new category in the "Other..." field of the Site Submission Form, that category was dynamically added to future entry forms. Links to the independently-located sites were then added to the students' Survey Evidence page for inclusion in final reports. For this second study, links were also added to the publicly-accessible search page:



Figure 5. Collaborative Search Page (minus sorting & searching fields)

Students can scroll through the list of located sites, click on the hyperlinked title, and browse the page. Alternately, they can enter in a person's name, a series of keywords, or a site title and the list will be reordered with matching sites at the head of the beginning (not shown in Figure 4.) By default, the sites appear in the order that they were entered with the first ten sites being drawn from the previous semester to provide a starting point.

The Heat Flow Analysis worksheet involved a critical analysis of the current house design. Students were asked to think about how the temperature would change over the course of the day:



Remember to: (a) label the temperature inside the house, (b) use arrows to indicate heat flow, (c) describe the things affecting heat flow.

Figure 6. Heat Flow Analysis Worksheet

They were given the approximate temperatures for the ground and the air at the different times of day.

The Redesign of the house could occur at any point during the activity. The instructions encouraged students to "Remember to label as many of the features as you can that relate to insulation, conduction, energy conversion, and heat flow.

The Design Report reiterates the initial instructions "to use your knowledge of heat, temperature, and energy to design a dwelling for use in the desert...You will have to decide what type of material to use for the walls of the dwelling, the shape and layout of the dwelling, the colors, what sort of windows, and so on. Don't worry about how much it costs or how hard it is to build. Design something that works in the desert climate and explain why your design would help keep people comfortable. All design decisions must be backed up with scientific evidence or principles." Partially completed sentences in the Design Report (e.g., "We decided to make our house out of...because...") prompt students to explain why they chose a specific material, color, or shape and to summarize the heat flow analysis using scientific principles, evidence from labs, and the Internet sites they visited in the Survey and Search phases of the activity.

Experimental Manipulation. A 3x5 factorial design was used that involved varying the example used in the Preliminary Design Worksheet and providing increased instructions on how to use the publicly-accessible search page.


      Period        Schematic           House+Schematic     Search 
Instr.       
        1                               
x                                       
        2           x                                       
x                   
        3                               x                   
x                   
        4           
x                                                           
        7                               x                   
x                   

Figure 7. 3x5 Factorial Design

The schematic drawing relating the amount of insulation inversely with heat flow is the inset in the house+schematic Preliminary Design example:


Figure 8a. Schematic Preliminary Design Example


Figure 8b. House+Schematic Preliminary Design Example

The periods that received additional instructions for accessing the publicly-accessible search page were told to visit the public page before trying to locate sites on their own. The other classes entered keywords on their own and only if they could not locate a useful site were they pointed to the publicly-accessible page.

Clinical Interviews. Between the Heat Flow Analysis and the Design Report eight students volunteered to participate in a 20-minute interview. The videotaped interviews explored the conceptual understanding of the heat flow worksheet. Subjects were asked to describe their dwelling design and the reasoning behind their heat flow analysis. Then they were presented with a graph with temperatures on the y-axis and the time of day on the x-axis. A function graphing the outside temperature over the course of the day was drawn on the graph. The subjects drew similar functions for the temperature of a well-insulated and poorly-insulated house. Then they were asked to draw a graph representing the temperature inside the house they had designed. Subjects were prompted to elaborate upon statements that appeared either contradictory or reflected some intuition about the scientific principles involved that was non-normative.

[ Abstract | Introduction | Methods | Results & Discussion | Conclusion | References | Bibliography ]