Problems in Developing an Alternative to the TOEFL

Abstract

An increasing number of programs and institutions have developed tests of English for academic purposes to be used in making admissions decisions at North American universities. It is not unreasonable for admissions officers to request information that will enable them to compare scores from a new and unfamiliar test with scores from the tests they have traditionally used. It is important, however, that the right questions be asked, and this is not always the case. What admissions officers frequently want is a conversion table calibrating scores from different tests, whereas the real question is not how well do two tests measure each other but how well does each test measure the construct of interest. Nevertheless, test scores are used as a basis for action, and it is important to provide decision makers with information that has applied utility until such time as satisfactory experience with the new test establishes its credibility. This article specifies a methodology for data collection, and compares appropriate statistical methods for data analysis including estimates of decision consistency, decision agreement, and shared construct relevant variance. The studies on which this article is based involved four groups of examinees (totalling 250) who wrote both the Test of English as a Foreign Language (TOEFL) and the Canadian Test of English for Scholars and Trainees (CanTEST).