Research Roundup 19, 3 (Spring 2003)
Scientifically Based Research
By Ron Beghetto
Most principals are aware that the No Child Left Behind Act (NCLB) of 2001 makes it mandatory for school leaders who depend on federal funding to select and implement programs that are based on scientific research.
The notion of scientifically based research in education is not new. Over the years, the federal government's efforts to make education a more scientific field has steadily progressed from the Cooperative Research Act of 1954 to the creation of the National Institute of Education in the early 1970s, later absorbed by the Office of Educational Research and Improvement (OERI). Two recent federal programs, the Comprehensive School Reform Demonstration Program and the Reading Excellence Act, emphasize research-based reforms.
In the latest reorganization of the U.S. Department of Education, OERI has been replaced by the Institute of Education Sciences (IES). One of the institute's first projects is the What Works Clearinghouse (available online at www.w-w-c.org/about.html), designed as a resource for educational decision-makers in selecting programs and practices based on scientific research.
Although there are no simple rules for determining whether the research base of a particular program or set of practices is scientific, useful guidelines and considerations have been developed that can assist principals and other school leaders in their decision-making.
The five documents reviewed in this Research Roundup offer insights into what is meant by scientifically based research, provide important considerations for school leaders when evaluating the scientific basis of practices and programs, and highlight the complexities involved in determining what programs and practices are most appropriate for a particular school.
Grover J. Whitehurst stresses the need for increasing the amount of scientific research in education.
The Comprehensive School Reform Program Office provides school leaders with guidance on the use of scientific research in initiating comprehensive reform efforts.
Stephen Raudenbush anticipates and addresses questions that educators might have about the nature and implications of scientific research in education.
Richard J. Shavelson and Lisa Towne provide a National Research Council primer on scientific research in education.
Evelyn Jacob and C. Stephen White edit a special theme issue of Educational Researcher on scientific research in education.
Whitehurst, Grover J. "Statement of Grover J. Whitehurst, Assistant Secretary for Research and Improvement, Before the Senate Committee on Health, Education, Labor and Pensions." Washington, D.C.: U.S. Department of Education, June 2002. Available online at http://www.ed.gov/offices/IES/speeches
In Senate testimony, Grover Whitehurst, director of the U.S. Department of Education's reorganized Institute of Education Sciences, focuses on the federal government's commitment to scientifically-based research in education, explains why the current emphasis on this type of research is important, and identifies key areas of research within the Department of Education that are in need of change.
Whitehurst explains that adherence to scientifically-based research will be a critical factor in the funding decisions and endorsement of programs that fall under the purview of NCLB. He argues that the research base surrounding key educational issues is thin to nonexistent, and that learning how to read is the only major program area that has "a substantial and persuasive research base." He claims that our lack of scientific knowledge in education "is masked by a 'folk wisdom' [that] employs unsystematic techniques doesn't demand scientific knowledge is inefficient, and...is hit or miss."
Whitehurst draws analogies between the current state of education and the premodern "folk practices" and "folk wisdom" of medicine and agriculture. He notes that significant advances in medicine and agriculture have resulted from a reliance on science and clinical trials, and asserts "there is every reason to believe that, if we invest in the education sciences and develop mechanisms to encourage evidence-based practices, we will see progress and transformation...of the same order and magnitude as we have seen in medicine and agriculture."
Comprehensive School Reform Program Office, Office of Elementary and Secondary Education, U.S. Department of Education. Scientifically Based Research and the Comprehensive School Reform (CSR) Program. Washington, D.C.: U.S. Department of Education, August 2002. 18 pages. Available online at http://www.ed.gov/offices/OESE/compreform/appendc.pdf
Administrators who seek federal funds to help defray the cost of implementing comprehensive school reform are required to select a reform program that "has been found, through scientifically based research, to significantly improve the academic achievement of students or has been found to have strong evidence that such a program will significantly improve the academic achievement of participating children."
This document provides school leaders with guidance on how to evaluate the scientific basis of reform strategies, methods, and programs by applying the following criteria:
Systematic and empirical. Research should involve careful planning, have a sound theoretical foundation, be grounded in observational and experimental data collected from multiple sources, and ensure that claims are supported by measurable evidence.
Rigorous data analysis. Analyses of data should utilize appropriate methods, address questions of interest, account for complexities of the data, and justify the general conclusions drawn from the study.
Reliable and valid data collection. Data should be collected professionally and
consistently to ensure that repeated measurements under similar conditions produce similar results, and that the collected data measure the outcomes they were designed to measure.
Strong research design. The research design should maximize the researcher's ability to answer the questions of the study and/or to test a hypothesis.
Results that allow for replication. Studies should present results with sufficient detail to ensure that replication and extension studies can be undertaken, and that the results are accessible and useful to practitioners.
Expert scrutiny. Research studies should undergo quality control from independent evaluators, such as peer reviewers from a scientific journal or an independent panel of experts.
Raudenbush, Stephen. "Scientifically-Based Research." U.S. Department of Education Seminar on Scientifically-Based Research, February 2002. Washington, D.C.: U.S. Department of Education. Available online at http://www.ed.gov/offices/OESE/esea/research/
A key section of this paper (one of several presentations at a U.S. Department of Education seminar on scientifically-based research that are available at the above Web site) addresses questions pertaining to scientific methods in educational research, including the following:
Do studies have to use random assignment to be considered scientific?
While random assignment is useful in establishing cause and effect because it controls for variables, a study can be considered scientific without it. However, the researcher must show that confounding variables were controlled and did not bias the outcome.
Are randomized studies possible in education?
Although randomized studies cannot answer every educational question, such studies are useful and should be employed with greater frequency. Raudenbush highlights several current and ongoing studies as examples, including the Tennessee class-size experiment, evaluations of school reform, and randomized studies of vouchers.
Does qualitative research play a role in making educational research more scientific?
Qualitative research serves the important function of providing vivid descriptions of how and why programs do and do not work.
How can insights be combined from various kinds of inquiry?
Drawing on medical research that has established the link between smoking and lung cancer, Raudenbush illustrates how non-experiments (surveys), true experiments (animal studies), and qualitative studies (examining lung tissue) "created a new consensus among scientists who had previously disagreed that smoking causes lung cancer."
Is there a risk in unrestrained enthusiasm for scientific research?
Raudenbush cautions against overestimating what science can offer education, explaining that "scientific work can inform but never replace the judgment of policy makers, practitioners, and parents." He favors rigorous peer review to help prevent the overselling of the promises of science.
Shavelson, Richard J., and Lisa Towne (Eds.). Scientific Research in Education. Washington, D.C.: Committee on Scientific Principles in Education, National Research Council, 2002. 204 pages. Available online at http://www.nap.edu.
This National Research Council report serves as a reasonably balanced and accessible primer on the topic of scientific research in education. The peer-reviewed report was generated by a diverse committee of prominent scholars who accepted the charge from the National Educational Policy and Priorities Board of the U.S. Department of Education to examine the nature and scope of scientifically-based research in education and to consider how a federal agency can support and sustain such research.
Several themes germane to educational research are put forward in this report. By carefully considering these themes, educational leaders will be in a better position to understand the nature of scientific research in education and determine the scientific basis of their present school programs and practices.
General scientific principles. The authors argue that these six general principles characterize all scientific research, including scientifically-based educational research.
- Poses significant questions that can be investigated empirically;
- Links research to relevant theory;
- Uses methods that permit direct investigation;
- Provides a coherent and explicit chain of reasoning;
- Replicates and generalizes across studies; and
- Encourages professional scrutiny and critique.
Accumulation of knowledge. The accumulation of scientific knowledge takes time, develops from diverse methods, relies on multiple studies across varied contexts, and complies with the norms and evidentiary standards of a scientific community. Conclusions regarding the causes and effects of any particular program or practice can rarely, if ever, be made on the basis of a single study.
Defining a scientific study. The design of a study is not sufficient for considering whether or not it is scientific. Rather, the scientific nature of a study is determined by its alignment with the general scientific principles outlined above and its adherence to the rigorous, self-regulated norms of a scientific community.
The nature of education. Because education is a complex, diverse, and value-laden field, it is important to carefully consider the unique features of a particular school context when determining whether theories and findings from research studies will meaningfully generalize to meet specific needs.
Jacob, Evelyn, and C. Stephen White (Eds.). Theme Issue on Scientific Research in Education. Educational Researcher 31, 8 (November 2002). Available online at http://www.aera.net/pubs/er/toc/er3108.htm
A recent theme issue of Educational Researcher included reactions from several prominent educational researchers, representing a variety of approaches and perspectives, to an article written by three key contributors to the National Research Council's report on Scientific Research in Education. In that article, Michael Feuer, Lisa Towne, and Richard Shavelson assert that the recent federal requirements for scientifically-based research create an opportunity to consider means of improving educational research. They presented four arguments in support of developing a "scientific culture of educational research" as a means for enhancing the scientific basis of educational research:
- Nurturing and reinforcing a scientific culture of educational research is a critical task for promoting better research.
- Scientific culture is a set of norms and practices within an ethos of honesty, openness, and continuous reflection.
- Individual researchers and research institutions have the responsibility for developing such a scientific culture.
- A federal educational research agency and the American Educational Research Association (AERA) can and must play crucial leadership roles in fostering, supporting, and protecting a scientific culture among their grantees and members.
In their response, James Pellegrino and Susan Goldman see diversity within educational research as a strength and declare that "educational research is often better served by the multidisciplinary, researcher-practitioner team approach." They explain that one of the most important criticisms of educational research is that it often lacks a meaningful connection to practice. They assert that educational researchers must include practitioners in their research community so as to better understand and address problems of practice.
David Berliner focuses on the complexities inherent in conducting research in education. For example, he notes that teachers' characteristics affect and are simultaneously affected by student and environmental characteristics. He also observes that changes in social and cultural environments can render findings from prior scientific research irrelevant.
Frederick Erickson and Kris Gutierrez explain that focusing only on "what works" can blind educators to important aspects and unanticipated outcomes of education processes. The authors admonish educational researchers and the consumers of research to develop a critical and realistic stance toward science. They stress the need for explicitly recognizing the complexity inherent in conducting educational research and the importance of multiple methods and perspectives as a means for promoting "positive educational change."
Elizabeth Adams St. Pierre criticizes the NRC report's rejection of postmodernism and the diversity of approaches included within that category. She argues for the need "to keep educational research an open field of play in which science does not obstruct but enables the proliferation of knowledge."
Ron Beghetto is an assistant professor of teacher education at the University of Oregon. He has conducted research on data-based decision-making, educational evaluation, and teacher development.
|
