THE IDEA BEHIND EBAPS
Table of contents
2. Critique of other epistemological assessments
3. Design principles for a new epistemological instrument
We want an instrument optimized for probing the epistemological stances of students taking introductory physics, chemistry, and physical science. Three popular surveys seem like promising candidates. One is Schommer's (1990) Epistemological Questionnaire (EQ), which is designed to apply to all disciplines and a broad range of age groups. The others, aimed specifically at college and advanced high school physics students, are Redish et al.'s (1998) Maryland Physics Expectation survey (MPEX) and Halloun & Hestenes' (1998) Views About Science Survey (VASS) In their respective communities, these surveys have brought unprecedented attention to epistemological issues. However, they contain important flaws. Schommer's EQ accurately probes students' epistemological stances toward physical science only to the extent that epistemological stances are stable beliefs or traits or theories that don't depend heavily on context, disciplinary or otherwise. As Hammer & Elby (2001) argue, this assumption is problematic. Consequently, we want an instrument that “works” whether or not students' epistemological stances depend on context. MPEX satisfies this condition in some respects but not in others. Also, by design, it probes not only students' views about knowledge and learning, but also their non-epistemological, course-specific beliefs about how to get high grades.
For these reasons, we designed a new survey, the Epistemological Beliefs Assessment for Physics Science (EBAPS). In section 2, we critique Schommer's EQ and Redish et. al.'s MPEX. These criticisms, which apply to other multiple-choice epistemological assessments currently in the literature, provide guideposts for designing a new instrument. Section 3 discusses how we formulated and validated EBAPS. In section 4, we critique EBAPS, acknowledging the ways in which it falls prey to some of our criticisms of other instruments.
In this section, we critically examine Schommer's Epistemological Questionnaire (EQ) and Redish et al.'s Maryland Physics Expectation survey (MPEX). We single them out for criticism solely because they are well-known in their respective communities and because our arguments apply equally well to other multiple-choice epistemological instruments.
EQ's strength—and in our view, its weakness—is the generality of its items, such as
(Other questions focus more specifically on school and learning, but without specifying a disciplinary or other rich context.) This generality might be problematic, for the following reason. As science educators and teachers, we want to know the extent to which students see scientific knowledge as certain versus tentative and evolving, among other issues. However, a student's reaction to the “death and taxes” item gives us accurate insight into her view of scientific knowledge only if students have stable beliefs or theories about certainty—beliefs that apply just as well to science as they do to everyday events or whatever else the student has in her head when she responds to the item. Hammer & Elby (2001) argue that this assumption is problematic. This assumption is plausible. For instance, as Hofer shows, students thinking about chemistry view knowledge as more certain than students thinking about psychology do. Even within a specific discipline, people's view of certainty might fluctuate. For instance, we are quite certain that the Earth is round (as opposed to flat), but quite uncertain about whether life exists on Mars. Hammer & Elby show that other contextual factors might also matter. [3]
Similar criticisms apply to the movie item. A student's response gives us precise insight into her view of scientific knowledge only if the student possesses a general tolerance or intolerance for uncertainty that permeates her view of knowledge in various disciplines. In other words, the movie item assumes that epistemological stances are traits; But epistemological stances, like many personality characteristics, might be less monolithic. For instance, we know a calm person who always becomes agitated and angry when she spends time with a certain annoying relative. Similarly, someone who dislikes ambiguity while relaxing at the movies on Saturday night might revel in uncertainty (after a good sleep) when debating a complex policy issue.
In summary, the extent to which Schommer's EQ accurately probes students' epistemological stances toward physical science depends on the extent to which epistemological stances are stable, context-independent beliefs or traits. We would prefer an epistemological instrument that works reasonably well even if students' epistemological stances turn out to incorporate disciplinary and other contextual dependence.
MPEX, like EQ, asks subjects to rate their agreement/disagreement with brief statements. But the statements refer specifically to physics, and in many cases, to introductory physics courses. Examples include
Even if students don't possess context-independent beliefs about the coherence of knowledge, these items probe whether they see physics knowledge in their course as coherent or piecemeal. However, many MPEX items conflate students' beliefs about knowing and learning with course-specific expectations and goals. For instance, MPEX tallies as “unsophisticated” a student who agrees with
Unfortunately, in fast-paced pre-medical physics courses that emphasize rote application of algorithms, a student may agree with this statement even though she knows that understanding physics involves insight and creativity. See Hammer's "Ellen" (1989) for a case study.Ellen would probably also agree with
MPEX underestimates that student's epistemological sophistication. Similarly, a student's answer to
probably reflects her goals and time constraints just as much as it reflects her views of learning. In summary, many MPEX--by design--taps into students' expectations and goals regarding the course, as opposed to their purely epistemological stances. The point of the above comments is that a student's expectations and epistemology can be out of alignment; see Elby (1999).[5] That's why teachers and researchers who want to focus on epistemology need another instrument.
In this section, we discuss the design heuristics underlying EBAPS. Then, we outline how the assessment is scored, and how we did validity testing.
The four authors started with an extensive literature review of epistemology research, reading papers and discussing most of them during weekly meetings. Synthesizing other researchers' ideas, we arrived at the following guiding principles:
Multiple dimensions. Our items fall into five epistemological dimensions, discussed below. As Schommer argues, it's productive for researchers to analyze students' epistemological stances in terms of dimensions. See Hofer and Pintrich for a full interpretive review of this issue. Importantly, experts disagree about whether these dimensions are reified cognitive structures inside students' heads or merely analytical categories for researchers. [6]
Multiple item types. EQ, MPEX, and many other assessments rely solely on Likert-scale “agree/disagree” items. But. We use three different item types:
Disciplinary specificity. Following MPEX and other discipline-specific instruments, EBAPS items relate directly to science and science learning, focusing on physical science. That way, whether or not epistemological beliefs depend on discipline, our assessment probes epistemological beliefs regarding physics and chemistry.
No `obvious' answer. Because physics instructors often preach about the real-life applicability of physics, alert students know they are supposed to agree with the MPEX item, “Learning physics helps or will help me understand situations in my everyday life.” We attempt, probably with mixed success, to pose items that students do not perceive as having an obvious sanctioned answer.
Rich contextualization: Hammer & Elby (2001), diSessa (1985), and others argue that epistemological “beliefs” are not always explicit, articulate, and consciously-held. Some epistemological knowledge, they argue, is implicit and inarticulate—more like procedural knowledge than like beliefs. For instance, according to Hammer (1994), few students hold conscious beliefs about the truth or falsehood of
Most students have never pondered this abstract issue. But their implicit “opinions” can be probed with closely-related items connected directly to students' experiences—items about which students have an opinion or at least a gut reaction. For instance, consider this EBAPS item:
(b) A small number of longer questions and problems, each of which covers several facts and concepts.
(c) Compromise between (a) and (b), but leaning more towards (a).
(d) Compromise between (a) and (b), favoring both equally.
(e) Compromise between (a) and (b), but leaning more towards (b).
Students have opinions (or at least gut reactions) about which exams are fair tests of their understanding. These opinions sheds light upon their (perhaps unarticulated) stances about the abstract issue of coherence vs. pieces. Of course, even if a student does hold a conscious belief about “knowledge in physics consists of many pieces,” her responses to our "best exam" item (and related items) sheds light on that belief. [7] The point is, we designed EBAPS to accurately probe students' epistemological stances concerning physical science whether or not epistemological beliefs depend on context (disciplinary and otherwise), and whether or not some epistemological knowledge is implicit (rather than belief-like or theory-like).
Above, we discussed the difficulty of probing students' purely epistemological beliefs, as distinct from their expectations about a particular course, their goals as learners, their own study habits, and so on. Our validity testing focused on this issue. Specifically, after a making two sets of revisions based on pilot subjects and informal feedback, we got about one hundred local community college students [8] to complete our assessment and write down their reasons for responding as they did to each item. [9] We then coded the responses looking for non-epistemological content.
It is instructive to review an item that got invalidated:
Several subjects who agreed wrote that, when they first encounter a hard new concept in a fast-paced class, they often “accept and move on,” but then go back later and try to make sense of it after they have more background. So, their agreement stems not from epistemological naivete, but from reasonable survival and learning strategy. The new version of the item may do a better job of getting at students' views about the coherence:
This kind of validity testing has the ability to pinpoint
issues deserving further study. For instance, in response to our
item “When it comes to learning science, memorizing facts is
extremely important,” some students wrote that it depends on
whether they're taking biological or physical science.
Apparently, some students hold explicitly discipline-specific
stances towards this issue.
3.3. What about reliability?
When developing instruments such as EBAPS, researchers often do reliability testing. Specifically, to make sure the items within a given subscale all probe the same beliefs, researchers refine or replace items that do not correlate with the others. We are not using this technique, for a principled reason. We don't want to assume that each subscale corresponds to a stable, consistent belief (or set of beliefs). For instance, consider these two items:
If both items pass further validity testing, and if students'
responses correlate poorly (e.g., if most students agree with both
items, even though the favorable repsonse ot the second item is
disagreement), it's not necessarily because
one of the questions is “bad.” It might be because
students are neither principled constructivists nor principled
absorptionists. For this reason, the EBAPS subscales should be
viewed as targets for instruction, not as categories of beliefs
residing in students' heads.
Indeed, if students' epistemologies consist not of "beliefs" but of
more fine-grained cognitive resources whose activation depends on
context, then we expect
students to disagree with themselves, so to speak, about different
items in the same subscale. Again, consider the two Nature of
Learning items listed above. The question about students' own
ideas
might trigger the idea that knowledge is something built up (an idea
they've abstracted from real-life experiences constructing knowledge),
and hence the idea that relating scientific concepts ot their own ideas
is productive for
learning. The lecture question, by contrast, might trigger the
idea that knowledge is "transmitted stuff," and hence, that really
clear transmission is sufficient for learning. As discussed in a cognitive
theory/practice paper, it's likely that students "have" the idea of
knowledge as transmitted stuff and
the idea of knowledge as built-up stuff; and which idea gets activated
in a given moment depends on contextual cues. By allowing
students to disagree with themselves within a given subscale — i.e., by
not considering this disagreement on its own to indicate
"unreliability" in the EBAPS items — we enable EBAPS to probe a range
of contexts relevant to learning physical science.
In section 2, we critiqued other epistemological instruments. In this section, we show how EBAPS fails to escape completely the criticisms we raised, despite our best efforts. Future revisions of EBAPS will alleviate some of these problems. Other problems with EBAPS (and EQ and MPEX) are probably a fundamental part of any multiple-choice instrument designed to probe a complex set of beliefs. For this reason, we think that EBAPS and other such instruments are best used in addition to interviews, case studies, and other methodologies that probe students' beliefs more deeply.
Major problems with EBAPS include the following:
Teasing epistemology apart from expectations. Our validity testing suggests that it's extremely difficult to write items that probe purely epistemological beliefs in nearly all respondents. For instance, even though
passed our validity tests, it's likely that some students answered “no” largely because they've never taken a challenging science course. In other words, some of our items probably trigger students' expectations about their classes and their epistemological beliefs, in ways invisible to the student herself, and hence, invisible to our validity testing. Deeper interviews would alleviate this problem, but not vanquish it entirely. Furthermore, even our “cleanest” items typically elicited non-epistemological reasoning from one or two subjects (out of the 50 or so).
Teasing beliefs apart from goals. Our item
is meant to probe students' views about the efficacy of effort. But a student could disagree simply because he doesn't want to put effort into his difficult physics class, and "convincing" himself that effort wouldn't make a difference anyway helps him rationalize his actions.
Inferring students' sophistication. We count as "sophisticated" agreement with
But a student could agree for unsophisticated reasons, such as believing that learning to think scientifically is no harder than learning to write cursive—it's just a matter of raw practice with no deeper cognitive change.
Inviting stock responses. An early version of EBAPS contained the item, “When it comes to learning science, memorizing facts is extremely important.” But since physics teachers often preach about the evils of “memorizing,” many students know that they're supposed to disagree. For that reason, we changed the wording (and for independent reasons, flipped the valence) to
Even so, some students probably know that they're supposed to agree.
[1] Hofer and Pintrich (1997) raise some of these issues.
[2] Using a Likert scale, students rate their agreement or disagreement with each item.
[3] A philosopher being treated for angina is quite certain, in all relevant senses, that her heart exists and pumps blood. But in an abstract academic discussion, the same philosopher may argue convincingly that scientific theories about the cardiovascular system are not certain. The philosopher's response to Schommer's item can and should depend on whether she's in a practical or an academic mindset.
[5] Some EQ items, such as “Everyone needs to learn how to learn,” also tap into values and goals.
[6] For instance, Schommer talks in terms of reified factors, while Hammer (1994) explicitly argues that his “axes” are not reified cognitive structures corresponding to stable beliefs.
[7] If a student's belief about pieces vs. coherence does not correlate with her beliefs concerning practical issues such as the textbook and exams, then that abstract belief is “empty,” not useful for predicting and explaining students' behavior.
[8] Drawn from six separate community colleges in northern California, these subjects were ethnically and socioeconomically diverse, judging by feedback obtained from the professors and by data about the student populations of those colleges. We did not collect data about ethnicity or SES.
[9] Each subject wrote out their reasoning for half the items in our 38-item survey.
diSessa, A. (1985). Learning about knowing. In E. L. Klein (Ed.), Children and Computers (Vol. 28, ). San Francisco: Jossey-Bass.
Elby, A. (1999). Another reason that physics students learn by rote. American Journal of Physics. Physics Education Research Supplement, 67(7 SUPP1), S52-S57.
Halloun, I., & Hestenes, D. (1998). Interpreting VASS Dimensions and Profiles. , 7(6), 553-577.
Hammer, D. (1989). Two approaches to learning physics. The Physics Teacher, 27(9), 664-670.
Hammer, D. (1994). Epistemological beliefs in introductory physics. Cognition and Instruction, 12(2), 151-183.
Hammer, D., & Elby, A. (forthcoming). On the form of a personal epistemology. In B. K. Hofer & P. R. Pintrich (Eds.), Personal Epistemology: The psychology of beliefs about knowledge and knowing . Mahwah, NJ: Erlbaum.
Hofer, B. K., & Pintrich, P. R. (1997). The development of epistemological theories: Beliefs about knowledge and knowing and their relation to learning. Review of Educational Research, 67(1), 88-140.
Redish, E. F., Saul, J. M., & Steinberg, R. N. (1998). Student expectations in introductory physics. American Journal of Physics, 66(3), 212-224.
Schommer, M. (1990). The effects of beliefs about the nature of knowledge in comprehension. Journal of Educational Psychology, 82(3), 498 - 504.
Return to EBAPS home.