| |
Research Roundup 19, 2 (Winter 2002-03)
Data Analysis for School Improvement
By Larry Lashway
Larry Lashway is a research analyst and writer for the ERIC Clearinghouse on Educational Management at the University of Oregon.
The public school system has always been somewhat ambivalent about the role of statistical data in education. On the one hand, schools generate an abundance of information. Principals, wearily confronting a steady flow of forms to fill out, know the list all too well: lunch counts, attendance records, test scores, discipline referrals, and dozens more.
On the other hand, the system has never been quite certain what to do with all this information. Traditionally, school leaders have satisfied their data obligations by filing the required numbers in the appropriate bureaucratic pigeonholes and moving on to the next task. Seldom have they been expected to actually use the data in making decisions.
Increasingly, however, state-mandated accountability systems and the recently reauthorized ESEA have forced schools to consider ways that data can be used for guiding school improvement. Moreover, the new legislation requires schools to disaggregate scores by ethnicity, disability, gender, socioeconomic level, migrant status, and English proficiency. Clearly, schools will need the capacity to efficiently extract and report a wide range of information and use it wisely.
Finding the right approach may not be easy. Told to be "data-driven," some school leaders may launch a single-minded push to raise test scores by any means at hand. Others may stray into a public relations approach that emphasizes spin control over substantive improvement.
The five documents reviewed in this Research Roundup find that schools benefit when leaders use the data to challenge their intuitions and assumptions. Defining "data-driven" simply as "the consistent use of objective information to enhance human judgment," the authors promote the use of data not just to satisfy legal requirements but also to focus action to improve student learning.
The Education Commission of the States examines the data practices of exemplary districts.
The American Association of School Administrators explains how data can be used to make better decisions.
Thomas Jandris offers principals a primer on making assessment a part of the school's culture.
Barbara Clements provides guidance on establishing automated data systems.
Edie Holcomb explores the human side of data use.
Education Commission of the States. Data-Driven Decisionmaking. Denver: Education Commission of the States, 2002. Available from: Education Commission of the States, 700 Broadway, Suite 1200, Denver, CO 80203-3460. $6 plus postage and handling for three-piece series; ask for GP-02-10. 303-299-3600; fax 303-296-8332. Web site: www.ecs.org. Available online at www.ecs.org/clearinghouse/35/52/3552.pdf.
This issue brief from the Education Commission of the States is one in a series aimed at helping educators come to grips with the challenges of the new ESEA accountability requirements. Based on a study of schools using exemplary data practices, it provides a brief but well-grounded overview.
First, the paper makes it clear that data are not restricted to test scores. Schools can also draw on demographic statistics, such as gender, ethnicity, and number of years in the district, as well as curriculum content, instructional interventions, and "perception" data, such as teacher beliefs about assessment or community attitudes about standards. Connecting these different types of data permits informed judgments about which students are benefitting from which approaches.
Second, data can be used for a wide variety of purposes, including tracking student
progress, guiding teachers' professional development, deciding on instructional and curricular
interventions, creating and assessing school improvement plans, and allocating district resources.
Third, data should be used throughout the district. At the school level, principals can
make sure data use is embedded in everyday practices, provide appropriate professional development for staff, and continually engage teachers in discussion about the data and their implications.
In the exemplary districts discussed in this report, student populations are characterized by low income, high mobility, and poor test scores. Their success in using data to improve instruction demonstrates that schools can move from "fixing the blame" to "fixing the problem."
American Association of School Administrators. Using Data to Improve Schools. Arlington, Va.: American Association of School Administrators, 2002. Available from: American Association of School Administrators, 1801 N. Monroe St., Arlington, VA 22209. Free publication. 703-528-0700; fax 703-841-1540. Web site: www.aasa.org. Available online at www.aasa.org/cas/UsingDataToImproveSchools.pdf.
Used correctly, good information leads to smarter decisions, defuses the emotion in controversial issues, and provides fuel for meaningful dialogue within the educational community. This report from AASA outlines a process for reaping those benefits.
Success begins with a clear sense of purpose. The data alone are meaningless unless linked to participants' understanding of what the school is trying to accomplish and how it plans to accomplish it. For that reason, leaders should discourage reflexive or simplistic responses to single
measurements.
Any data point should generate numerous interpretive questions before action is taken. How does this score compare with previous scores? Do different demographic groups score differently? Do results reflect recent changes in the instructional program? Do the results suggest the need for changes? The goal is not to make everyone a data technician but to create an understanding that intelligent use of information can move the school closer to fulfilling its mission.
Because public reporting requirements ensure that school data will attain high visibility, the report suggests three strategies for cultivating community involvement and understanding:
- Engaging the public in dialogue can give schools insights into what information is most meaningful to the community.
- Crafting a thoughtful communications plan will make sure that parents and public get accurate information in a timely way.
- Working with the media will get the school's message before the public.
The report concludes with a few words of caution: Don't make quick decisions without consulting all the key players; don't exalt data for its own sake; and prioritize the indicators that are most important. Most of all, remember to continually remind people of the big picture, the purpose behind the data and the benefits they will bring.
Jandris, Thomas P. Data-Based Decision-Making: Essentials for Principals. Alexandria, Va.: National Association of Elementary School Principals, 2001. Available from: National Association of Elementary School Principals, 1615 Duke St., Alexandria, VA 22314-3483. $19.95 members; $24.95 non-members; 10 percent discount if ordered from Web site; Order Number EDBDM. 800-386-2377; fax 800-396-2377. Web site: www.naesp.org.
The current interest in assessment data is motivated by the demands of accountability. But the real power of data is unleashed only when assessment is infused throughout a school's culture. This monograph by Thomas Jandris, prepared under the sponsorship of NAESP and the Educational Research Service, explores the leadership challenges for principals.
The first step is recognizing the need for a comprehensive approach. Limiting data use to the annual state-wide test will fail to harness the full power of assessment. The report borrows from the Japanese kaizen philosophy that produces quality through continuous incremental improvements at multiple points along the way, rather than a single "inspection and correction" at the end of the process.
One of the principal's key responsibilities is to build sound assessment practices into the school's culture. Principals create the right conditions when they establish a supportive climate, provide time for assessment, train teachers to use assessment data when planning instruction, and state clear expectations for performance.
Assessment scores can have multiple meanings, depending on who is interpreting the results. To maintain coherence, leaders must anchor the information to these key questions: What do we want students to know and be able to do? What kind of assessment will tell us how students are doing on these criteria? How well are they performing? What does the assessment suggest about ways to improve?
Answers to such questions will come only through continuous dialogue, debate, and discussion. The author recommends using the Annenberg Institute's "Inquiry Cycle," which focuses improvement efforts into a continuous six-step process:
- Establish the desired outcomes;
- Define the questions;
- Collect and organize the data;
- Make meaning of the data;
- Take action; and
- Assess and evaluate actions.
In addition to providing principals with a strong conceptual background to guide their use of data, the report offers numerous concrete examples, a long list of resources, and a glossary of assessment terms.
Clements, Barbara S. Building an Automated Student Record System: A Step-by-Step Guide for Local and State Education Agencies. Washington, D.C.: National Forum on Education Statistics, 2000. ED 442 475. Available from: ED Pubs, P.O. Box 1398, Jessup, MD 20794-1398. Free publication. 877-433-7827. Available online at www.nces.ed.gov/pubs2000/building.
With the new ESEA requirements, keeping track of the data may turn out to be as big a challenge as gathering it. Not surprisingly, many schools have turned to computerized storage-and-retrieval systems. This booklet from the National Forum on Education Statistics provides administrators with a quick but comprehensive introduction to the subject, outlining 12 steps, from setting goals to reporting the data.
Automated systems carry several advantages, including cost savings, accuracy, flexibility, quicker retrieval, and easier reporting. However, computerization also heightens the potential for data proliferation. Therefore, before schools make the commitment to automate, they should determine what information is important, how it will be used, and who will use it. In most cases, schools will be looking for information that satisfies accountability requirements, provides feedback on the overall health of the school, or guides instructional improvement efforts.
A major concern with any record system, especially an automated system, is security and privacy. As many users of e-mail have discovered, electronic information can quickly spread beyond the intended users. Because federal privacy regulations require that schools provide access only to those with a recognized "need to know," policies should indicate who has access to various kinds of records, what security procedures will be used to prevent inappropriate access, and what penalties will be imposed for violations.
At the same time, schools should develop procedures to ensure that students and their parents can access personal records, as required by law. Finally, procedures should be established to discard outdated information.
Although this booklet does not provide comparisons or other technical guidance on specific software or hardware systems or vendors, it will help school leaders ask the right questions when the time comes to make a purchase.
Holcomb, Edie L. Getting Excited About Data: How to Combine People, Passion, and Proof. Thousand Oaks, Calif.: Corwin Press, Inc., 1999. 162 pages. ED 433 373. Available from: Corwin Press, Inc., 2455 Teller Road, Thousand Oaks, CA 91320. $29.95 plus shipping and handling. 800-818-7243; fax 805-375-1700; e-mail order@corwinpress.com. Web site: www.corwinpress.com.
Data analysis inevitably revolves around numbers, which many people regard as cold, abstract, or even dehumanizing. But as Edie Holcomb argues in this book, human hopes, fears, and passions are never far beneath the surface.
In the end, Holcomb notes, effective data use is less a technical problem than a cultural problem, and it needs to be approached on that basis. Drawing on her experience as staff developer and consultant, she provides a road map for leaders faced with the challenge of "getting people onboard."
Teachers have many reasons for resisting data analysis. They may lack the time or the right kind of training, or they may fear that the results will be used against them. The key is to help teachers connect the data to their passion. One way of doing this is through the school mission statement. Reminded of the mission, teachers can proceed more readily to two questions: What data do we currently have that can help measure our success? What additional data could we use?
Once teachers begin considering data that might help them make decisions, they are often overwhelmed by the possibilities. Holcomb offers three strategies to avoid data paralysis. Her preferred approach is constructivist, relying on intensive dialogue on questions such as: What evidence would demonstrate fulfillment of the mission? Can we find more than one indicator for a particular goal? What data will verify our assumptions about student learning? What data might help resolve smoldering issues in the school?
Holcomb notes that readiness varies, but that the important point is to start somewhere. Schools not ready to make the commitment to a constructivist approach can focus on compliance (dealing only with the data required by external agencies) or even minimalism (using whatever data happen to be at hand).
While Holcomb concedes at the start that her book is neither highly technical not comprehensive, it provides a detailed road map for getting teachers involved with, and motivated by, data. Readers will find not only a coherent conceptual framework but also useful advice on a variety of practical issues, from finding time to displaying data in a user-friendly way.
|
