Analyzing Dependability and Bias in WDCT and DSAT using Many-Facet Rasch and G-Theory

Pragmatic testing depends on a variety of factors that can impact its dependability. This study intended to examine these factors using a two-phase approach. The first phase examined the impact of test methods, items, raters, and test-takers' characteristics on the variance in pragmatic test scores using generalizability theory, and the second phase explored potential rater bias using the Many-Facet Rasch model. Two test types, including a Written Discourse Completion Test (WDCT) and a Discourse Self-Assessment Test (DSAT), were administered to ۱۱۰ English language students (۹۸ female, ۱۲ male) aged ۱۷-۲۴ at Vali-e-Asr University of Rafsanjan. Four raters scored the WDCT by using a standardized rubric developed by Lui (۲۰۰۴). The DSAT was self-assessed by test takers based on the same rubric. The findings revealed no significant difference between the WDCT and DSAT test types. However, items and the interaction between items and test takers emerged as substantial contributors to the variance of the scores. This highlights the importance of item calibration and rater training to mitigate bias in pragmatic testing. Finally, the implications were discussed.

کلیدواژه ها:

Pragmatic testing ، Generalizability theory ، Many-Facet Rasch model ، WDCT ، DSAT