Formative Assessment for College Readiness: Measuring Skill and Growth in Five Key Cognitive Strategies Associated with Postsecondary Success

This study reports the preliminary results from a field test of the College-readiness Performance Assessment System (C-PAS), a large-scale, 6th-12th grade criterion-referenced assessment system that utilizes classroom-embedded performance tasks to measure student progress toward the development of key cognitive skills associated with success in college. A sample of 1,795 students completed C-PAS performance tasks in English and mathematics at 13 New York City high schools in grades 9-12 during Fall 2007. The performance tasks were derived from construct maps and “task shells” designed to elicit the key cognitive strategies. Teachers administered the tasks to students and scored the tasks using standardized scoring guides. Preliminary analyses using Item Response Theory (IRT) yielded evidence that C-PAS measures the acquisition of college readiness cognitive thinking skills in both math and English. The study is significant because it suggests that cognitive strategies important to college readiness can be measured discretely and within separate subject areas. Additionally, the study suggests that complex performance assessments can be utilized to systematically contribute useful information on student performance to help improve student learning. This is important given the current search for ways to address some of the limitations of current large-scale testing methods and systems. Formative Assessment for College Readiness 3 Formative Assessment for College Readiness: Measuring Skill and Growth in Five Key Cognitive Strategies Associated with Postsecondary Success Introduction The proportion of high school graduates pursuing postsecondary education has increased consistently over time, yet evidence suggests that many admitted students are unprepared to succeed in college-level instruction (Greene & Foster, 2003). The 2005 National Education Summit on High Schools termed this problem the “preparation gap” (American Diploma Project, 2006). While 67% of high school completers pursue some form of postsecondary education immediately after high school (National Center for Education Statistics, 2005), 30% to 60% of these students require remediation in math or English, or both (California State University System, 2007; Conley, 2005). These shortcomings cut across all racial and ethnic lines (Venezia, Kirst, & Antonio, 2004), but are most pronounced among first-generation college attendees, a group that overly represents low income and minority students. This design of C-PAS seeks to address the “preparation gap” by providing feedback on the degree to which students are developing key cognitive strategies essential for success in entry-level college courses. Descriptions of high school instruction paint a consistent picture of classrooms in which students complete prescribed tasks that require little cognitive engagement, often in order to prepare for state tests that may not align well with college readiness (Angus & Mirel, 1999; Brown & Conley, 2007). In an accountability-driven era, few high school teachers appear to have the time or inclination to develop student-thinking skills. As a result entering college students often show difficulty retaining, understanding, transferring, and applying much of the knowledge they have been taught, a phenomenon termed “fragile knowledge syndrome” (Perkins, 1992; Perkins, Jay, & Tishman, 1993; Perkins & Salomon, 1989). Formative Assessment for College Readiness 4 College faculty nationwide, regardless of the selectivity of the institution, expressed near universal agreement that most students arrive unprepared for the intellectual demands and expectations of post-secondary environments (Conley, 2003). College instructors appear to accept the fact that many incoming students may not have retained content knowledge taught to them previously, and those who teach entry-level courses appear to be willing to reteach as new material much of what has been taught previously in high school (Conley, et al., 2008; Conley, McGaughy, Cadigan, Forbes, & Young, 2009). However, they also expect students to make inferences, interpret results, analyze conflicting source documents, support arguments with evidence, solve complex problems that have no obvious answer, reach conclusions, offer explanations, conduct research, engage in the give-and-take of ideas, and generally think deeply about what they are being taught (National Research Council, 2002). Students who have little prior experience developing these cognitive strategies struggle when confronted with content knowledge they have not retained well that they are now expected to process and manipulate in much more complex ways. Researchers have analyzed high school transcripts and found that rigorous academic preparation as represented by the titles of high school courses taken is the most significant explanatory variable for persistence to college graduation (Adelman, 1999; Bedsworth, Colby, & Doctor, 2006). A different approach is to analyze the content of college courses and then determine what should be occurring in high school courses to align with what will be encountered in college courses. Research in this area has identified key attributes of college readiness, most notably a series of metacognitive strategies and essential content knowledge (Conley, 2005). The C-PAS assessment model is based on elements of this research, most importantly, the notion that effective college preparation must include development of key Formative Assessment for College Readiness 5 cognitive strategies and that those strategies must be developed while studying essential content knowledge. Objectives The purpose of this study was to field-test the College-readiness Performance Assessment System (C-PAS) in order to determine the validity of its conceptual design and constructs and to evaluate its ability to measure five Key Cognitive Strategies (KCS): problem solving, research, interpretation, reasoning, and precision with accuracy. The College-readiness Performance Assessment System (C-PAS) was designed to enable teachers to monitor the acquisition of the KCS through rich content-specific performance tasks embedded into the curriculum. Postsecondary preparedness is the reference point for this criterion-based measurement system. The five Key Cognitive Strategies (KCS) are always learned and practiced in the context of challenging content knowledge. The variance in tasks is limited by a focus on the five KCS, which are measured through common scoring guides. The study employs itemresponse models to report the preliminary results from the psychometric analysis of the field test data. Performance assessment, also known as authentic assessment, seeks to measure student knowledge or skills through products that result from their engagement in and completion of a task rather than their responses to a series of test items. Performance-based assessments have undergone study in a variety of settings over the past 20+ years with varying results. They were used extensively in the early 1990s during the first wave of educational standards and were found to be difficult to use for high-stakes accountability purposes (Koretz, Stecher, & Deibert, 1993), interest in performance assessment is reviving as the limitations of current large-scale assessment methods are being recognized, particularly the lack of connection between tests and Formative Assessment for College Readiness 6 classroom instruction and the emphasis such tests place on recall and simple application items that tend to gauge lower-level cognitive functioning. The concern is that this type of testing is driving classroom teaching in the wrong direction, away from complex thinking and toward simple recall without understanding. Performance assessment does theoretically have the potential to provide more meaningful feedback to students and teachers (Cohen & Pecheone, 2008) in ways that inform teaching behaviors because the assessments themselves are deeply embedded within the instructional process. Further, performance tasks allow students to demonstrate much more complex and diverse thinking than do multiple-choice item tests, and they provide opportunities for students to actively apply skills and knowledge to real life situations rather than simply selecting the “right” answer from among several choices or in the context of an artificial problem or situation (Cohen & Pecheone, 2008; Wilson, 2005). Theoretical Framework The C-PAS model is grounded in three theoretical frames: a dispositional-based theory of intelligence, cognitive learning theory, and competency theory. A dispositional or characterological view of intelligence builds on incremental theories of intelligence that believe intelligence is malleable and recognizes that ability is a continuously expandable repertoire of skills, that through increasing efforts, intelligence can grow incrementally (Bransford, Brown, & Cocking, 2000; Costa & Kallick, 2000). The second conceptual frame derives from emerging cognitive learning theory, referred to as the “New Science of Learning.” This contemporary view of learning asserts that people construct new knowledge and understandings based on what they already know and believe. Perkins (1992) condenses this fundamental understanding into a single sentence: “Learning is a consequence of thinking. Retention, understanding, and the active Formative Assessment for College Readiness 7 use of knowledge can be brought about only by learning experiences in which learners think about and think with what they are learning” (p. 8). Competency theory provides the final element of the conceptual frame and serves to bridge between developmentally appropriate student cognition and assessment (Baxter & Glaser, 1997). Competency theory is guided by the expert-novice literature and suggests that novices (students) benefit from models of how experts approach problem solving, especially if they receive coaching in us