Students will be selected based on a number of criteria,

Determining a Higher Education Readiness Component for TAKS A Research Proposal
Table of Contents
I. Introduction and Overview of Proposal
II. College Readiness Component Proposal
III. Higher Education Focus Group
IV. Follow-up Study
V. Conclusion Introduction

Background Information The Student Assessment Division of the Texas Education Agency (TEA) convened a committee in the fall of 2000 to provide guidance and to make suggestions for the implementation of the college readiness component of the new statewide testing program, the Texas Assessment of Knowledge and Skills (TAKS), as mandated by Senate Bill 103. The intent of the legislation is to use performance on the test to assess not only a student’s level of academic preparation for graduation from a Texas public high school but also the student’s readiness to enroll in an institution of higher learning. The Higher Education Readiness Component Task Force met four times during the 2000–2001 academic year. Through extensive discussion, review of educator survey results received from Texas institutions of higher education, and analysis of placement tests and freshman curriculum data collected from a representative sample of two- and four-year colleges and universities, the committee identified a number of challenges in establishing the link between performance on TAKS and readiness for college-level work. • There is little, if any, agreement as to what constitutes readiness for college and success in college or on college-level work. However, the Texas Academic Skills Program (TASP) represents a state standard for college readiness. • There are many different opportunities for students to receive post-high school instruction, including but not limited to public colleges and universities, private colleges and universities, junior colleges, and community colleges. • The level of instructional rigor and the diversity of course offerings vary among institutions of higher learning in the state of Texas. Page 2 2 During its May 1, 2001, meeting, the task force reviewed proposals for assessing college readiness using TAKS. The committee agreed that an amalgamated proposal incorporating elements of several suggested options would be the most beneficial for determining a TAKS college readiness component. Overview of College Readiness Component Proposal The college readiness component proposal includes research investigations that will be used to determine a college readiness scale score for the exit level TAKS. The research design includes a modified contrasting groups methodology study and a data correlation study of performance on TAKS in relation to performance on the TASP test, the ACT assessment, and the SAT I. The proposal and implementation guidelines are outlined below. The academic rigor and broad content range of the new TAKS assessment suggest that it will be a stronger potential indicator of success on college-level work than is the current testing program, the Texas Assessment of Academic Skills (TAAS). Consequently, a strong relationship between performance on TAKS and performance on post-high school work is anticipated. The goal of the proposed research is to investigate this relationship. A sample of 2,000 second semester college freshmen (1,000 per subject area in English language arts and mathematics) will be administered the TAKS assessment during the spring of 2003. This sample will represent both two- and four- year public college students and will be as racially and demographically balanced as possible. The sample will not include students enrolled in developmental education courses. Using a modified contrasting groups methodology, the performance of this group, who have demonstrated a measure of college readiness by their continued matriculation, will be compared to the performance of Texas public high school juniors who take the same test in April 2003 . The second segment of this proposal will correlate TASP, ACT, and SAT I test result data with data from TAKS testing for the same student population. Legislation establishing the TASP test was passed in 1987 and is located in section 51.306 of the Texas Education Code. The test is designed to ensure that students attending public institutions of higher learning in Texas have the academic skills necessary to perform college-level work. TASP test results are used by public two- and four-year colleges and universities to make placement decisions and to determine developmental education needs. In addition to their reliance on the TASP test, Texas public colleges and universities have a long tradition of incorporating national college entrance examination results into their admission and placement decisions. Although the ACT and SAT I are just two measures among many that colleges and universities use to assess student preparation, these tests are used extensively and have an established reputation for providing reliable data about student readiness for college. The research design calls for TASP, ACT, and SAT I results for Texas students who are public high school juniors in the spring of 2003 and those who are juniors in the spring of 2004 to be correlated with the corresponding TAKS Grade 11 exit level results from the spring 2003 and 2004 test administrations. TAKS and TASP, TAKS and ACT, and TAKS and SAT I expectancy tables will be established via regression methodology, and correlations will be determined. Other correlations among the tests will also be explored including test item and performance objective analyses. Page 3 3 Data collection and analysis for the contrasting groups study will be completed during the fall of 2003; data collection and analysis for the performance data correlation study will be completed in fall 2003 for those juniors who take the TAKS in spring 2003 and have taken the TASP, ACT, or SAT I by the following fall. These data will be provided to the Texas Higher Education Coordinating Board for use in establishing, prior to the spring 2004 exit level TAKS administration, a scale score that indicates readiness for enrollment in an institution of higher education. College Readiness Component Proposal Contrasting Groups Methodology The more rigorous exit level TAKS will have a broader content range than the current exit level TAAS and should therefore be a stronger potential indicator of success on college-level work. Consequently, a strong relationship between performance on TAKS and performance on post- high school work is anticipated. The goal of this proposed research is to investigate this relationship. The spring 2003 TAKS testing instrument will be the primary data collection device for the first segment of the study. A representative sample of second semester college freshmen will respond to the same testing instrument that is used for the exit level TAKS program in the spring of 2003. Test results for the two populations will be compared using a modified contrasting groups methodology. The research design being proposed is taken from “standard setting” methodology and is a modification of the area of work typically referred to as “contrasting groups” (Berk, 1976; Zieky & Livingston, 1977). In the contrasting groups design, two populations of students are identified, those seen as “masters” and those seen as “novices.” Both of these populations of students are given the same assessment instrument, and their performance is compared. A clear and empirical relationship is determined which shows how the masters compared to the novices on the assessment. The point where the two populations diverge is taken as the “optimal” cut score on the instrument that would most often classify the masters and the novices correctly. For purposes of the current research, the assessment instrument will be the exit level versions of TAKS in English language arts and mathematics. The high school population will be first time TAKS test takers (all students in Grade 11) in the spring of 2003. The college population will be comprised of a sample of second semester freshmen at public two- and four-year postsecondary institutions in Texas. Page 4 4 Analysis of the results will yield two different distributions of raw scores for each TAKS subject, one each for the high school and college populations. These distributions will be on the same assessment (TAKS) and can be directly compared as indicated in the figure below. Figure 1. Example of Contrasting Groups Results The advantages of such a research design are numerous. First, no definition of success needs to be identified. The comparisons are empirical, based on the performance of the two groups of examinees. Second, the sample selected for the college population should include a wide range of returning freshmen with diverse background characteristics. Third, there are no psychometric or statistical manipulations necessary. The test is given to two groups, and the results show how they compare. A sample of 1,000 students per subject area is the suggested minimum for the study in order to ensure reliable data. Distributions of test scores based on fewer students may be too unstable to warrant reliable conclusions regarding comparability of performance between high school and college-level test takers on TAKS. Students will be selected based on a number of criteria, including enrollment in college level courses, socioeconomic background, ethnicity, and gender. Upper-level administrative personnel will be contacted to promote student and institutional participation. In addition, incentives, possibly including monetary compensation, may be provided to encourage student cooperation. The confidentiality of the test instrument will be maintained by administering the test to the college sample on the same schedule or after the TAKS exit level high school exams are administered. Whenever possible, college and university testing centers will be utilized to ensure consistency and test security. NCS Pearson, TEA’s contractor for the current TAAS program and for the development of the TAKS program, will be involved in the development and administration of the contrasting groups research design. Working in conjunction with TEA, NCS Pearson will design a survey form to be completed by each college student participating in the research. This survey will ask for information about the student that will provide the link between the student answer documents and the student background information. NCS Pearson will not collect personal, confidential, or proprietary information from college students; the link between test scores and background information will be independent of identifiable student names or numbers. For example, NCS High School Population College Population TAKS Score Scale Page 5 5 Pearson could provide a scannable answer sheet such that one side of the document collects all of the relevant background information while the other side collects student responses. This document could be “pre-gridded” with college number and with the student name blank. Only the school official would know the identity of the student taking the test and for whom the background characteristics are being provided. In such a way, strict confidentiality of student information can be maintained. NCS Pearson will provide a final report on the contrasting groups study to TEA. This report will include a description of the participating schools, a comparison of performance between the high school and college populations, and commentary regarding anything discovered during the research that was unanticipated or of additional interest. TEA will forward this information to the Texas Higher Education Coordinating Board for use in determining what scale score on TAKS might best indicate readiness to enroll in an institution of higher learning. TASP, ACT, SAT I, and TAKS Performance Data Correlation Study The second component of the research proposal looks at college readiness using TAKS scores in relation to performance on the TASP test and to performance on national standardized college entrance exams. Currently, students who attend public institutions of higher learning in Texas must take the TASP test unless they are exempted based on their ACT, SAT I, or TAAS scores. Colleges and universities use TASP test results to make placement decisions and to determine developmental education needs. Of the 189,766 students who took the TASP test between September 1998 and August 1999, slightly more than half (99,101) were in the 15–19 age group. Approximately 61% of the 184,144 students who graduated from Texas public high schools in 1999 chose to take the ACT and/or the SAT I. The 113,807 individuals who took these exams represent a surprisingly broad spectrum of students. The majority of four-year Texas colleges and universities, as well as many junior and two-year colleges, require that prospective students take the ACT or SAT I. Although these tests are just one measure among many that colleges and universities use to assess student preparation, the ACT and SAT I have an established reputation among college personnel and are routinely used for both admission and placement purposes. The proposed research study will look at the TASP, ACT, and SAT I score results for Texas public high school students who take these tests in the spring, summer, and fall of 2003 and compare them to the spring 2003 Grade 11 exit level TAKS scores for the same student population. In addition, student test results from the spring 2004 Grade 11 exit level TAKS administration will be compared to TASP, ACT, and SAT I scores from the spring, summer and fall administrations of these tests. One methodology proposed for determining correlations between TAKS scores and TASP, ACT, and SAT I achievement is to create TAKS/TASP, TAKS/ACT, and TAKS/SAT I expectancy tables and perform regression analyses or, equivalently, contingency table analyses to establish predicted TASP, ACT, and SAT I scores from given TAKS scale scores. Glasnapp and Poggio (1996) used this same methodology in establishing cut score equivalents for the current Grade 10 exit level TAAS test and the TASP test. For each TAKS score, the analysis would indicate the Page 6 6 percentage of students with that score who also achieved a specified score on one of the other tests (TASP, ACT, or SAT I). TEA will coordinate the TAKS/ACT and TAKS/SAT I studies, and the Texas Higher Education Coordinating Board will coordinate the TAKS/TASP analysis. Other statistical correlations will also be explored. Because the ACT and TASP are more content- based than the SAT I, subject area score correlations may be examined. Regression analyses will help to determine how predictive student performance on TAKS will be in relation to TASP, ACT, and SAT I results, particularly in the subject areas of mathematics and reading skills. A study designed to compare TASP test items to TAKS test items will be incorporated as an ancillary activity to provide a more thorough understanding of content variations between the TASP and the TAKS. This investigation may also include a comparison of ACT assessment standards to TAKS exit level student expectations to enhance interpretation of the ACT/TAKS correlation study data. Data from the correlation study will be compiled by TEA and Texas Higher Education Coordinating Board staff and will be used by the Texas Higher Education Coordinating Board to determine a college readiness scale score prior to the spring 2004 administration of the Grade 11 exit level TAKS. Higher Education Focus Group A focus group made up primarily of representatives from two- and four- year Texas public institutions of higher learning, as well as some representatives from private higher education institutions and public high schools in Texas, will be convened by the Texas Higher Education Coordinating Board and the Texas Education Agency during the 2002 spring semester to give feedback to the Higher Education Readiness Component Task Force regarding the implications of the TAKS higher education component plan for colleges and universities. The group will also be asked to provide insight about performance expectations for entering college freshmen and will serve as a link to the higher education community, providing college and university personnel with ongoing updates regarding the development and implementation of the TAKS higher education component. Follow-up Study of College Readiness Indicators An ancillary demographic study of the relationship between performance on TAKS and performance in college will provide valuable follow-up information about how successfully Texas public high school students are making the transition from high school to college. College readiness and success indicators such as course selection, placement, program rigor, grades, and retention rates will be correlated with TAKS performance results to furnish additional information about the relationship between TAKS scores and college readiness. Student Assessment will work with the Texas Higher Education Coordinating Board and the TEA Public Education Liaison Information Office to obtain college student performance data. The study will begin in the fall of 2004 when the first group of TAKS graduates enters college. Page 7 7 Conclusion The Higher Education Readiness Component Task Force is charged with providing guidance and making suggestions for the implementation of the higher education readiness component of TAKS as mandated by Senate Bill 103. The committee has proposed a research study to determine a higher education readiness score for the exit level TAKS examination. The research design includes a contrasting groups analysis of college student performance compared to high school student performance on the TAKS, as well as a TASP/TAKS, ACT/TAKS, and SAT I/TAKS performance data correlation study. The Student Assessment Division at TEA will be responsible for managing the research process with the Higher Education Readiness Component Task Force and the Texas Higher Education Coordinating Board providing assistance as needed. Data collection will begin with the 2003 TAKS administration. The Texas Higher Education Coordinating Board will establish, prior to the spring 2004 exit level TAKS administration, a scale score indicating readiness for enrollment in an institution of higher education. Students who achieve this score will be exempt from the requirements of TASP. A follow-up study will be initiated in the fall of 2004 for the purpose of obtaining additional information about the relationship between TAKS scores and college success measures indicative of academic preparation for college. Since a scale score indicating higher education readiness will not be established until the spring of 2004, some avenue must be provided for those students who are sophomores in 2003 to qualify to register for dual credit courses using their TAKS results without taking TASP or an alternative test. After the Texas Higher Education Coordinating Board has established a higher education readiness scale score for the spring 2004 TAKS administration, it will be necessary to provide a permanent way for students to qualify for dual credit courses before their junior year begins. The Texas Higher Education Coordinating Board will establish a temporary qualifying score on TAKS that will allow sophomores in 2003 to gain permission to register for dual credit courses and a permanent means for sophomores in 2004 and thereafter to get permission to register for dual credit courses using their TAKS results. Page 8 8 References Contrasting Groups Study Berk, R. A. (1976). Determination of optimal cutting scores in criterion-referenced measurement. Journal of Experimental Education, 15, 4–9. Zieky, M, J., & Livingston, S. A. (1977). Manual for setting standards on the basic skills assessment tests. Princeton, NJ: Educational Testing Service. TASP/ACT/SAT I and TAKS Performance Data Correlation Study Glasnapp, D., & Poggio, J. (1996). Examining Equivalents for the TAAS and TASP. The University of Kansas: Center for Educational Testing and Evaluation. The Texas Higher Education Coordinating Board and National Evaluation Systems, Inc. (2000). TASP Technical Summary 1998–1999. Texas Education Agency, Division of Performance Reporting. (2000). Snapshot 2000. Texas Education Agency, Division of Performance Reporting. (2000). Results of College Admissions Testing in Texas for 1998–99.