A many-facet Rasch measurement of randomness in peer-assessors

In secondlanguage performance assessments where assessors are employed to assess language learners’ written products, assessors commit errors which may threaten the reliability of the ratings awarded to the products. Randomness, or inconsistency, is one of the persistent errors which affect language learners’ performances. In the present study, we used a many-facet Rasch measurement approach to detect and measure randomness among peer-assessors. To that end, 39 peer-assessors participated in a two-hour training programme to learn how to assess 20 five-paragraph essays MA students wrote in an Advanced Writing course. The peer-assessors used a 6-point analytic rating scale to assess the essays on 15 assessment criteria. We employed Facets, a computer software package, to analyse the rating data for this study. We ran a many-facet Rasch measurement analysis which included peer-assessor, students’ essay, and assessment criterion as facets. The results of Facets analyses showed that not all peer-assessors assessed the essays consistently. Some peer-assessors were internally consistent when assessing the essays, but they were not consistent with other peerassessors. These peer-assessors tended to assess the essays in a different order than did the other peerassessors. The findings of this study suggest thatextensive training is requiredto make peer-assessors consistent.