Period 1: mud houses (not many houses made of mud), houses in Arizona, desert, insulation, heat energy, windows, dry, land, windows, energy, build, construction, air conditioning, structure, dome.
Period 2: mud house, wood house, straw house, desert climate, materials, building, insulation, heat, night, pueblos, windows, adobe, flammable, dome, *energy, *conservation.
Period 3: house (pictures, real estate, construction companies), building, materials, heat flow, adobe, insulation, Indian houses, desert house, building, colors & shapes, windows, insulators & conductors, *energy, *conservation.
Period 4: desert house, insulation, windows, glass, double pane, energy efficient, foundation, building materials, insulation materials, energy conservation.
Period 5: [no keywords generated]
U F Keywords Very Somewhat Not At All No Rating Total 1 1 insulation 11 2 14 5 32 2 3 architecture 3 0 7 4 14 3 2 desert 2 5 7 1 15 4 7 dome 2 1 2 1 6 5 13 insulated 3 1 0 0 3 windows 6 23 Arizona 1 0 1 0 2 Houses 7 24 climate 1 0 0 1 2 8 25 solar panels 1 0 1 0 2 9 30 energy 0 1 1 0 2 conservation 10 15 heat 1 0 2 0 3 conservation
The creativity and inconsistency in users' approaches to processing
has driven research in HCI for years. For example, Dennis Egan in his
evaluation" studies with the SuperBook browser (cf.
Landuaer, 1991, p. 64), noted that, "It appears to be reformatting
- rearrangement and highlighting - of the page, or perhaps the search context
in which it was found, [that] caused one group of users to notice and absorb
the information where the others passed it over."
Similarly, it is difficult to determine whether the non-uniform rating of
sites observed in the first experiment is due to context, the interactional
disposition of the user, or some combination.
During the search phase, 44 of the 66 groups (~67%) found at least one Internet site that they located independently and thought useful enough to save to their Netbook. Percentages are rounded to the nearest whole number:
In the final reports, sites located during the search phase were cited
50% more frequently than the sites visited during the Survey Evidence phase.
The sites in the Survey Evidence phase came from the Networked Evidence
Database (NED) and were selected by the activity designers as relevant to
the project. These NED sites were cited by 16 of the 66 groups (~24%) as
opposed to individually-located Internet sites that were cited by 25 of
the 66 groups (~37%) (3 groups discarded). Of those students that cited
evidence in their final reports, 28 of 34 or approximately 82% cited either
NED evidence or other (individually-located evidence). Only 18% of the
that cited evidence cited BOTH types of evidence.
Only 8 of the 66 final reports (approximately 12%) did not reflect structural aspects of the evidence found in the Survey Evidence phase. The influence of the sites presented in this early phase of the activity persisted in the form of dome-shaped mud dwellings similar to the Native American adobe structures that were compared with straw and wood buildings. Of the eight groups that developed designs that did not involve a dome-shaped or mud dwelling, only one group formulated their design based on a lab conducted during the semester (e.g., an energy conversion lab.) This group used water to store the heat energy during the day and release it slowly at night. The other seven groups developed innovative designs based on information that they located individually on the Internet. The average grade for these seven groups was 72.1 vs. the class average of 67.7 (rounded to one decimal place) or 6.5% above the average.
Collaborative Search Experiment. The change in activity structure from the first to second running of the intervention successfully expanded the problem definition for the majority of the students. This expanded problem definition helped students generate a wider range of keywords and locate useful information. The addition of a Revised Design Worksheet and Heat Flow Analysis Worksheet had little impact on the selection of features considered for the dwelling (e.g., windows, size of rooms, orientation of house, type of insulation, etc.) but did reveal internal conflicts within the students' explanatory framework for the temperature differences at different times of day. This cognitive dissonance occasionally resulted in a reevaluation of the design or reassessment of the explanatory mechanisms for describing heat flow. Initial designs reflected a strong bias towards the examples used to illustrate heat flow and towards the selected sites that were visited early on in the project.
Students generally retained the framework of their initial designs and used evidence as a "proof of existence" for features of a house rather than as an argument for selecting one design over another. Principled reasoning about the dwelling design came late in the process in either the Heat Flow Analysis worksheet or the Final Report. Students that did not use evidence to generate alternatives (i.e., the consumers) had better than anticipated success in searching for useful information but not in developing a well-structured argument.
From the first to second experiment, there was an increase of 23% (to 74%) in the number of groups that cited evidence in the final reports. In addition, citations of NED evidence increased 44% (to 68%) while evidence located during the search phase decreased by 9% (to 28%). There was an increase of 16% (to 83%) in the number of groups that located sites that they thought were useful. The following table shows the breakdown by period for sites saved during the search and cited in the final report. Percentages are rounded to the nearest whole number:
Of those students that cited evidence in their final reports, 27 of 44
or approximately 61% (down from 82% in the first experiment) cited either
NED evidence or other (individually-located evidence). Thus, the number
of students citing BOTH types of evidence increased from 18% to 39%.
The 50% increase in citations of independently-located sites over selected NED sites in the first experiment also has multiple possible explanations. This finding may confirm the hypothesis that information located in the preliminary design phase is used to generate options while information found during later stages is used to support existing designs. An alternative explanation comes from looking at the "type" of evidence encountered during these two phases. The material presented during the Survey phase (part of the preliminary design) consisted primarily of examples of different houses (e.g., straw, wood, and mud). The material located independently by students typically dealt with specific structural aspects of the dwelling such as roofs, windows, or foundations. The Enertia site (see Figure 12) located by students during the search phase of the first experiment was an exception containing information about thermal inertia and delta T.
This site was used in an activity that preceded the second experiment
and was subsequently recalled by students during the search episode of the
second experiment. However, it was not sited as frequently as the mud, straw,
and wood sites though many of the designs reflected the heat flow envelope
and radiant floors modeled at this site. The fact that citations by groups
of both NED and independently located sites increased from 18% to 39% of
the groups most probably reflects the increase in the number of sites
on the Survey Evidence page as opposed to a change in how information
To analyze groups, categorical assignments were made based on the manner in which evidence was incorporated and arguments presented in the final reports. The purpose of this categorization was to provide an indication of the potential value of using categorical variables to describe the design/search process. The intent was not to report a tightly controlled experimental finding. These categories were assigned based on the following criteria:
Strategizers: tried to produce what they thought the teacher wanted (e.g., two citations of Internet evidence, reference to two labs, etc.)
Conceptualizers: framed their designs using a top-down framework with scientific principles being presented up front and the details falling out of the principled framework.
Consumers+ (aggregation): incorporated whatever information they encountered into their designs in an ad hoc fashion.
Consumers= (replacement): replaced their design with some piece of information they found but did not critically analyze.
Experimenter++ (adds principles): added principles to support design alternatives.
Experimenter=> (refines ideas): refined ideas to support integrated design components.
Aggregating the consumer and experimenter subtypes yields a composition
of 53% (consumers) and 16% (experimenters). The high ratio of consumers
to other types of IDs may be attributed to the complexity of the design
Innovative Designs: Only 9 of the 69 final reports (approximately 13%) did not reflect structural aspects of the evidence found in the Survey Evidence phase. This result is consistent with the first experiment. Interestingly, 6 of the 9 innovative designs were created by groups that reflected a consumer-based interactional disposition. However, the source for these innovative designs came from a Web site that was discovered in one period and shared through the exchange of keywords within and across periods.
NED 0 1 1 0 0 0 Other 0 0 1 0 1 0 Shared 0 0 2 2 1 0 Totals: 0 1 4 2 2 0
In an excerpt from the final report, we can see an attempt to develop
a mechanism for heat flow.
Part of our home is constructed underground. This helps us because the termpature stays more constant compared with the outside temperature. We have inserted vents that allow cooled air to lift up into the house forcing the hot air to rise to the top and escape through more vents.
These vents located at ground level force the cool air into the house. The hot air rises and is forced out through the top vents. Vents are obviously to keep the house cooler during the day. This is becuase heat energy passes through air easier than it does through glass.
This reasoning emerged in the final report and although the scientific language is non-normative (e.g., escape, cooled air [lifting] up) the line of reasoning is potentially generative.
A less successful example of developing a mechanism using principled reasoning can be seen in the House Of Water (see Figure 16). The design was copied directly from an existing Internet site.
The design itself is complex using several adjacent rooms, a water
system, and a solar turbine. The principles related to heat storage and
thermal mass that justify the use of plastic filled with water as an
are not developed as we can see from an excerpt from the final report:
The material we are using for our house is straw and plastic filled with water because both straws and plastics are very good insulators.
The plastic walls and small windows keep out the sun's heat and retain warmth at night.
Even when students begin to account for the surround and the ground temperatures, problems can emerge because of the complexity of the issues and the difficult mapping that must occur to apply labs to real-world situations. For example, in the House of Mirrors (see Figure 17) the students raise the house off the ground to avoid heat transfer between the ground and the house.
The final report indicates that they have applied knowledge from the
Scattering Lab to select white as a color for the house and shiny silver
for the roof.
The color we chose for our house is white because it absorbs the least amount of light and heat and will keep the house the coolest. The roof is shiny silver so as to reflect sunlight overhead. We made it shiny silver and not mirrors because in one lab we learned that shiny silver works better as a reflector than a mirror does.
However, they do not consider the problem that a metal roof may serve as a good conductor in addition to scattering the sun's light energy. Similarly, they do not consider the possibility that the temperature of the surround may have a greater effect on the dwelling than contact with the ground.
The general trend to support designs with principles late in the design process would be expected of strategizers realizing that they need to support their designs with principles as a requirement for the activity. The exception to this rule would be expected for conceptualizers but is hard to support empirically from the data because of the difficulty with assessing the presence or absence of conceptual reasoning. A finer-grained definition of categories may be needed to capture the subtleties that distinguish the different phases of the design process.
Grades: Experimenters and conceptualizers had the highest average grade score while consumers trailed by approximately 10 percentage points.
The ranges for Consumers' grades (58-94 for Consumer+; 61-94 for
were greater than for the other groups (76-97 Strategizer) with the
(85-99) and Experimenters (82-97 Exp++; 84-97 Exp=>) reaching the higher
levels more consistently.
Citations, Queries, Saves: Experimenters and conceptualizers entered almost twice as many queries as consumers. Experimenters that added principles used evidence located during the search phase twice as frequently as experimenters that refined their ideas. Consumers had the lowest rate of NED evidence usage in their final reports. Conceptualizers saved the most number of sites that they thought useful to their Netbook.
The increase in the number of sites that were deemed useful and saved
during the search phase (up 16% to 83% in the second experiment) could be
the result of an expanded problem definition, the accessibility of the
Search Page, and/or the effectiveness of generating keywords collectively.
The fact that experimenters and conceptualizers entered twice as many queries
as consumers reinforces the link between interactional dispositions assigned
from analyzing the final reports and the independently measured search
Similarly, conceptualizers rarely used search evidence - an expected result
if they were focusing more on the explanatory mechanisms in place of
based on evidence. Conceptualizers along with experimenters had the highest
average scores; something to be expected since the grading was based on
conceptual understanding, innovation, and justification.
>1 Citation/Query/Save: Strategizers used NED evidence almost twice as frequently as consumers. Experimenters that added principles used evidence from the search phase more than twice as often as strategizers, consumers, and experimenters that refined ideas. Conceptualizers rarely used evidence from the search phase.
Somewhat surprisingly, experimenters that added principles had twice
as many search citations as strategizers, consumers, and experimenters that
refined ideas. The aggregate style with which this type of experimenter
incorporated information could explain this phenomenon. The lower citation
rate by consumers reflects their tendency to adopt alternatives arbitrarily
without refining them or integrating them into a conceptual framework. The
lower citation rate for strategizers can be clarified by examining the high
rate of citation for NED-based evidence; a finding consistent with their
characteristic of incorporating more reliable and acceptable pieces of
Experimenters that refined ideas may have condensed their explanations or
developed innovative designs that relied more on principles than
Collaborative Search Page. Approximately 41% (28 of 69 groups) used either the search or sorting capability of the collaborative search page. An average of 20% of the groups searched for some specific student name, set of keywords, or category. 40% of the groups that used the collaborative search page were some type of consumer.
In the two periods (one and two) that encountered a particularly useful
site (e.g., Home Power: "How To Stay Cool In The Desert"), 50%
of the groups were looking specifically for the Home Power site using either
student's names or keywords (see Figure 22).
There was no discernible effect from the experimental condition for periods two, three, and seven. In fact, period two had a significantly lower usage rate for the CSP than the other periods: 27% vs. 40% to 54% for the other periods. However, the final design reports for those groups that did use the Collaborative Search Page in period two reflected a strong bias for, if not wholesale adoption of, the information located using the page.
Not surprisingly, all of the groups looking for the site were consumers.
Discounting that group of consumers, the proportion of strategizers using
the search page rises to 35% with the conceptualizers, consumers, and
leveling off as approximately 20% each.
A combination of the type of evidence, the stage of the design process, and the interactional disposition of the user are factors in how evidence is used. For example, consumers had the lowest rate of NED evidence usage. This finding, at first surprising, could be explained by the possibility that consumers best exemplify the tendency to justify decisions after making them. However, analysis of the usage statistics for the Collaborative Search Page reveals that consumers accounted for 40% of the activity. One possible inference is that consumers had already processed the early evidence, had located a relatively low number of useful sites, and were simply at the stage of the activity where they needed to find evidence to support their designs. From an information foraging perspective (Pirolli, 1996), the cost of accessing relevant material in the locality of the Collaborative Search Page (CSP) was much lower than seeking out information on their own. And in fact, the results from experiment #2 indicate that consumers were frequently targeting information located by other groups.
Leveraging Student Intuitions. Intuitions that students have about heat flow and energy conversion appear implicitly in the initial sketches and somewhat more explicitly in the final design reports. The following comments encountered in the final reports reflect various p-prims (diSessa, 1988) and facets (Clark, 1996) of principled reasoning. The purpose of categorizing these facets is a) to refine the categorization scheme used to classify interactional dispositions and b) to provide examples of the types of non-normative conceptions that students bring to the project. I provide examples of how these non-normative conceptions can be transformed into more normative explanations. Ultimately, students intutions may be used to provide different types of activities that have been successful at producing conceptual change in students with similar intuitions. The following comments are grouped into categories for the purpose of beginning such an analysis:
Containment & Escape2.06: "the advantage of having windows is, it will make the heat flow in the house easier so the "used air" has more openings to get out of."
3.08: "windows on our house will be one way mirrors because it will allow light to travel into the house but won't be able to escape but reflect off the mirrors and stay in the house."
3.14: "at night the desert cannot give back the energy and it is very cold."
4.02: "the heat that managed to make its way into our house will be trapped in the walls while trying to escape"
7.07: materials need holes to let heat in and out. "the heat is trying to get into the house"
Illusion Of Coolness1.12: marble is naturally cool, white reflects heat
4.08: things that feel cool keep things cool
7.01: white gives illusion of coolness
Invented Mechanisms For Heat Flow & Exchange2.05: small and low windows with black lines around the windows "because the sun will go toward the outside of the window. I got this idea from the baseball players (they put black around their eyes)."
3.05: "it takes longer for an object that is an insulator to heat up, but when it does heat up, it will retain the heat longer."
3.07: "heat flows the direction the sun is facing."
4.07: heat doesn't flow through thick things. "a good insulator is something that keeps you cold."
7.02: things that heat up faster have more heat energy
7.06: dense objects are good insulators
7.09: "In the daytime, heat will flow on the side of the house, run up the house, and off the roof."
7.10: "the pool water reflects sunlight and the house becomes cooler"
7.08: "we will not have too much furniture because that will not allow very much condensation of heat or cool in the desert homes."
Two of the most prevalent p-prims appear to be "heat flows out of objects" and the derivative p-prim "objects must be physically connected for heat flow to take place." The clinical interview with student 4.05 provides examples of both of these p-prims:
i: so how does the heat energy from the sun affect the house temperature?
s: because it's scattering first of all the rays are scattering most of them are coming into this general area (side of house) like from here to here and it's being absorbed by the house so it's heating the house.
i: but without changing the outside temperature too much?
s: uh huh. it definitely changes the outside temperature a lot. but see i'm kind of confused because see if the sun is coming up it's got to be getting warmer so this (the outside) will be increasing and this the ground temperature will be increasing. wait unless the cold air is coming in here (from air to ground). no, yeah.
i: which do you think will increase faster the temperature of the outside air...?
s: yeah the outside air?
i: faster than the ground?
s: yeah. the air will be heating the - basically the heat is going into the air raising that temperature. that will raise the temperature of the heat and the temperature of the heat will raise the temperature of the ground. (draws arrows from sun to air to ground to house) because the house is right on the ground.
i: so if you had to draw heat flow arrows...
s: [draws heat flow arrows] it would go from sun into air then "down" into the house then from the house into the ground. that's messed up if the ground is warmer than the air. oh i know it would be going from here to there from the ground to the house. the house might even be giving off a little energy.Another common p-prim is that heat can flow both out and in at the same time:
i: so you think it can go both ways from the air into the house and from the house into the air?
s: um hm.
i: can it go both ways or does heat flow just one direction?
s: i'm trying to remember the lab we did. oh! heat always flows out. out of an object. you know the lab we did using the good and poor conductors. and you just have to find a way to slow it down. but it flows out if its hot if its 90 degrees or whatever if its a pot and it's 90 degrees its always going to flow out of the pot.
i: you talked a lot about ventilation i'm just wondering if that's the same kind of principle as heat energy moving through insulation.
s: maybe it would like be the same. is heat circulating the same thing as heat going through insulation? i dont know, i don't think so because i mean the actual circulation is caused by like the cool air pushing the hotter air towards the ceiling towards higher up in the altitude. that's it. when the cool air gets inside the house it pushes the hot air to higher altitudes.
The intuition that heat energy gets "used up" or "trapped
in walls" can be converted into more normative explanations by linking
principles with concrete examples. During the clinical interviews many of
the students were able to develop explanations by referring to labs they
had conducted earlier in the semester and applying them to the novel
of describing heat flow at the different times of day. For example, one
group referred to the Coke and Potatoes lab where they had arrived at the
same principle for describing how a cold Coke warms up and a hot potato
cools down. They then proceeded to describe the heat flow between the house
and the surround at the different times of day in terms of similarity to
either the Coke or the potato.
This type of reformulation worked for helping students in the interviews reason about the functional representation of copies of their house that were either much larger or smaller in scale. To effectively solve this task students needed to refer to the Pulsing lab where they had added equal amounts of heat energy to different quantities of the same substance in this case water. Interestingly, even those students who were able to link the lab to the task at hand had difficulty determining which house would have more heat energy at the end of the day. For example, student 7.02 thought that the smaller house would have more heat energy because it had heated up faster. Frequently students can recite principles but do not understand the mechanisms that underlie the phenomenon that those principles explain.
The tendency for students to select principles that support their designs late in the design process in not necessarily undesirable. In fact, the heat flow analysis was intentionally placed before the final report so that students would begin to think about heat flow in terms of day-night cycles rather than the static view adopted by most students during the first experiment. Conceptualizing heat flow in terms of functions greatly helped the students that were interviewed develop a more coherent and integrated understanding of the relation between the various elements in their design. For example, group 7.03 was able to graph the function representing the temperature of the water relative to the temperature of the environment (sold line: environment; dashed line: water.)
This graphical representation helped them explain how the water would
work to both heat and cool the house at different times of the day.
The heat flow analysis worksheet presented students with an opportunity to critique their existing designs. This worksheet revealed the difficulty that students had in explaining how heat energy transfers from the sun to the house. The idea that heat transfer occurs through contact with some object provides the basis for many of the heat flow descriptions. However, problems arise when the "surround" is not included as an intermediary in the transfer process. This limited conceptualization of heat transfer may account for the tendency for students to disregard more principled pieces of evidence in favor of sites that describe specific structural aspects of the dwelling such as the windows or the roof. Performing a heat flow analysis or critique of existing designs as part of the problem structuring phase could help students develop a more principled approach to gathering information resources.
Qualification Of Results. The attempt to provide frames for analyzing the data reported here has a number of inherent problems. First, the fuzzy nature of interactional dispositions means that membership in a category is graded in addition to being dynamically redefined over the course of the activity. The categorization scheme does not reflect this fact. However, the purpose of analyzing the ID categories has been to discover the potential for using them to link behavior patterns in different phases of the activity in preparation for the next phase of this research project. Second, there are problems with the post hoc explanations that link behavior patterns. However, by laying out specific hypotheses about how evidence is used during the problem definition versus the detail design phase these problems are reduced to some extent. Further research is needed to determine if these explanations have strong emperical support. Third, the hierarchy of ID groupings is not clearly defined primarily because we are not sure whether it is better to approach design problems from a conceptual standpoint or to support design alternatives using principles after those alternatives have been generated. The probable outcome of this debate is that the top-down versus bottom-up approach depends on some combination of the student's ID and the task at hand.
Finally, the fact that the second experiment allocated more time (i.e., half a day) to the search activity can be disregarded as the cause of the increased rate of sites deemed relevant. We have shown that more time searching does not typically lead to the location of more relevant sites for a majority of students (i.e., there is a threshold for most students (Cuthbert, 1996)). And in reality students in both experiments ended up having similar time allocated to the search activity even though the activity plans differed slightly.