December 3, 2004 Abstract: Recent research has established that teacher effects on student learning are sizable, sometimes enough to match or overcome the effects of socioeconomic status. While research provides some glimpses of the elements that may contribute to these effects, few studies have looked comprehensively and systematically at measures of teacher quality that are related to teacher effectiveness in the classroom. Our interest is in studying the relationship between measures of pre-service preparation quality, individual teacher quality, teaching practices, and student achievement gains for early career teachers.
The study capitalizes on the development and validation of the Performance Assessment for California Teachers (PACT) assessments, a performance-based system for assessing pre-service teacher quality designed and piloted by 16 California universities. The PACT is a direct descendent of the National Board for Professional Teaching Standards, adjusted to assess pre-service teacher candidates and to take account of teaching issues (such as the teaching of English language learners) that are important in California. It includes a structured performance assessment and a regimen for scoring that has been honed over two years of pilot assessments.
For elementary teacher education graduates of a number of the PACT universities, we will evaluate performance and effectiveness using measures of pre-service teacher preparation experiences (components of and perceptions of preparation in both content and pedagogical training), pre-service measures of teacher quality (e.g. coursework grades, licensure test scores, supervisory ratings, and PACT scores), and measures of teacher effectiveness (as measured by student achievement growth in literacy and mathematics and, if additional funding is attained, as also measured by ratings and descriptive evidence of teaching practices).1
The sample will consist of approximately 200 elementary teacher credential candidates, half of whom will have completed performance assessments of their literacy instruction during student teaching, and the other half of whom will have completed parallel assessments of their mathematics instruction. Data on school contexts and student characteristics will be used to guide sample selection and to provide appropriate statistical controls. Multivariate analyses of the predictors of teacher effectiveness will be conducted. These analyses will also provide concurrent and predictive validity evidence about the PACT assessment as a measure of teacher quality.
With growing national interest in teacher quality, this research has the potential to inform state and higher education policy and practice with respect to teacher preparation, licensure, and accreditation.
I. Rationale and Background The goal of this research is to study the relationship between measures of preparation quality, individual teacher quality, and beginning teacher effectiveness, including student achievement growth. Our findings are aimed at informing two goals: 1) to improve the selection and preparation of teacher candidates; and 2) to guide policy decisions about where to invest resources to improve aspects of teacher quality that have a strong link to student achievement.
Teacher quality has come to national attention in part due to the requirements of the No Child Left Behind (NCLB) legislation that requires a highly qualified teacher in every classroom. A strong body of recent research evidence indicates that teacher quality is a strong contributor to student achievement (e.g., Sanders & Rivers, 1996; Rivkin, Hanushek, & Kain, 2001; Nye, Konstantopoulos, & Hedges, 2004). At the same time, there is disagreement over which aspects of teacher quality are most salient for student learning. In considering this question, some researchers have focused on available indicators of teacher quality such as teachers’ education, performance on licensure tests, and certification status (e.g., Ferguson, 1991; Goldhaber & Brewer, 2000). Darling-Hammond’s (1999) review of research on teacher quality and student achievement found that measures of teacher preparation and certification were often the strongest correlates of student achievement, even after controlling for student demographics. Her state-level analysis found that the proportion of well-qualified teachers (those with both certification and a major in their field) accounted for as much as 40% of the variation in student achievement, after controlling for such factors as student poverty and language background.
Other scholars (e.g., Kennedy, 2004; Wenglinsky, 2000b) have asserted that research should also consider other dimensions of teacher quality that are more difficult to capture, including measures of 1) the specific content of teachers’ training, 2) teachers’ beliefs about subject matter, pedagogy, and students and 3) teachers’ actual instructional practices. In this study, we propose to do exactly that. We will have available measures that will allow us to describe the content of teachers’ preparation, their performances as pre-service candidates, their perspectives on teaching, their perceptions of their preparation and induction support, their practices as new teachers, and growth in their students’ learning.
With the resources requested from Carnegie Corporation, we will be able to follow beginning teachers from the last year of their preparation into their first year of teaching, evaluating the connections between their preparation experiences, performance as pre-service candidates on traditional measures and performance assessments, and performance as teachers as evaluated by their students’ achievement. We also plan to conduct analyses of teachers’ practices in the classroom (also described in this proposal) if additional resources are secured. The resources for the full research design and for this request are outlined in the budget. Figure 1 provides a schematic for the research design, which is more fully described in what follows.
The research questions will investigate relationships among the aspects of the preparation program; the nature, intensity, and perceived quality of support during preparation and induction; measures of individual teacher quality; teacher practices; and student learning gains. The specific research questions include the following:
What are the relationships among aspects of teachers’ preparation in content and pedagogy and their performances on assessments of teacher knowledge and skill (e.g. traditional licensure tests; performance assessments)?
What are the relationships among different measures of candidate performance (e.g. grades, supervisory ratings, traditional licensure test scores, PACT scores)?
What are the relationships among aspects of teachers’ preparation experiences, their performances on assessments of teacher knowledge and skill, their practices during the first year of teaching, and student learning gains?
How are these relationships mediated by school contexts and student characteristics?
What is the relationship between the nature, intensity and perceived quality of support during preparation and induction (the first year of teaching) and student learning gains?
The unique context of this study is the PACT (Performance Assessment for California Teachers) Consortium, which presently consists of sixteen public and private California teacher preparation programs. The consortium has been working since the summer of 2002 to develop and implement a set of subject-specific assessments of teaching knowledge and skill. The PACT assessments build on efforts by the National Board for Professional Teaching Standards (NBPTS) and the Interstate New Teacher Assessment and Support Consortium (INTASC), which developed performance assessments for use with expert and beginning teachers. Like the NBPTS and INTASC assessments, the focus of the PACT assessments is on candidates’ application of subject-specific pedagogical knowledge; this focus is based on research indicating that effective use of such knowledge is related to successful teaching (Bransford, Brown, & Cocking, 1999; Darling-Hammond, 1998; Fennema et al., 1996; Grossman, 1990; Porter, 1988; Shulman, 1987).
The PACT assessments were developed in response to a California state mandate (SB 2042) that called for all teacher preparation programs in the state to implement valid and reliable performance assessments for use in making credentialing decisions. In 2002, teacher educators and directors from 12 teacher preparation programs formed the PACT consortium and began developing an alternative to the California Teaching Performance Assessment, the state-sponsored prototype. Since then, four more institutions have joined the PACT. These institutions include all eight University of California programs, four California State University programs (Sacramento State, San Francisco State, San Jose State, and San Diego State University), three programs at private institutions (Mills College, Stanford University, and the University of Southern California), and one intern program (San Diego City Schools). With modest philanthropic support and substantial in-kind contributions from each of the colleges/universities, the consortium piloted and scored portfolios in six credential areas in 2002-03: elementary literacy, elementary mathematics, English/language arts, history/social science, mathematics, and science. The assessments were then revised for 2003-04, piloted, and scored again. Over this two-year period, more than 1,200 candidates at the PACT institutions piloted Teaching Events in the six content areas. In addition, more than 250 teachers and teacher educators were trained to score these assessments in 2002-03 and 2003-04. (A recent technical report on the PACT is enclosed as an attachment to this proposal).
The study, which will examine the relationships between components of pre-service preparation, teacher performance on the PACT, other pre-service indicators of quality (grades, traditional licensure tests, supervisory ratings), and indicators of practice and effectiveness, will provide data that both explores the predictors of teacher practices and effectiveness and that examines the concurrent and predictive validity of the PACT assessment. In what follows, we describe research on the elements of teacher quality that inform our research design. In a subsequent section, we discuss the sampling design and measures we propose to use to examine these questions.