Psychometric challenges in developing a college admission test for Jordan

Social Behavior and Personality, 2001 by Alnabhan, Mousa, Harwell, Michael

In 1998, the Jordanian Council of Higher Education authorized the construction of a standardized aptitude test that would be used to assist colleges and universities in admissions decisions. This paper reports the results of a study that examined whether test items were operating as desired and path analyses that explored predictors of student performance for a highly selective sample of Jordanian students. Item analyses indicated that thirty percent of the items showed inadequate discrimination or inappropriate difficulty levels, and an additional nineteen percent of the items showed evidence of differential item functioning attributable to sex. The path analyses indicated that the strongest predictors of performance emerged for female students and included parental educational level and whether students attended a government-sponsored school or a private school. For males, the same predictive relationships were negligible.

The use of standardized tests in college and university admission decisions is well established in many countries but less so in others, including Jordan. In 1998, the Jordanian Council of Higher Education authorized construction of a standardized aptitude test that would be used to assist colleges and universities in admissions decisions. The rationale for the Council's decision lies in Jordan's dramatic population growth and the anticipated need for a skilled and well-educated labor force in the 21st century. The availability of an admissions test with strong psychometric properties and whose content is closely linked to the skills needed to succeed in college can play an important role in this process. The authors' charge was to investigate the psychometric characteristics of the test using data from a pilot study.

Although empirical investigations of new tests are fairly standard, two factors complicated this work: (1) The sample of Jordanian students used in the pilot study represented the top 5-15% of academically talented 10th and 11th grade students in Jordan, as nominated by their secondary (high) school, raising the possibility of restricted variation in the data. (2) The educational advantages that often accrue to males in Jordanian society made it likely that strong sex differences existed in test performance. Both of these factors could pose significant psychometric challenges to the test development process.

The authors investigated also the role of other variables that may affect test scores. One was school authority, which essentially reflects the organization sponsoring the school attended by students (government sponsored versus private schools). In Jordan, private schools are beyond the reach of many citizens because these schools have high tuition rates and fees, and most children from poor and middle-class backgrounds attend schools sponsored by the Jordanian government. As a result, school authority provides a rough measure of a student's socio-economic status, and it was hypothesized that this variable would affect test performance because of the large economic and social differences between the two types of schools.

In addition, the relationship between student achievement and some familybased factors was investigated. Earlier work had found a negative relationship between the achievement of Jordanian students and family size, and a positive relationship between achievement and parental educational level (Abu-Lebdeh & Ahlawat, 1997; Alnabhan, 1997). These findings led the authors to hypothesize that students from larger families would be less likely to perform well because, in Jordanian society, large families are often associated with lower levels of parental education. It was also hypothesized that parental education would positively affect test performance. For example, students from better-educated families may outperform those from families in which parents do not have much education, although this effect may be mediated by the academic success of students.

The purpose of this study was to examine the psychometric properties of the test to determine if they were adequate for test development to be continued (Other researchers and practitioners were responsible for ensuring that the resulting test closely matched the skill domains to be assessed). The authors were particularly interested in examining the effects of restricted variation in the test response data and the magnitude of sex differences. Secondary questions of interest included learning if school authority, father's education, mother's education, or family size affected test performance. Currently, no empirical investigation of the test has been undertaken, and these results will help to guide subsequent revisions.

The construction of the test began in 1999 with a national panel consisting of content experts in Math, Science, English, Arabic, and Measurement and Statistics. These experts were charged with writing items that assessed specified domains, for example, verbal skills. With the construction of 100 dichotomously scored items, each with five options, the panel felt that the test was ready to be piloted. Like most college entrance examinations, the test is intended to be normreferenced and is described as an aptitude test.