Abstract:
Of paramount importance in the study of cognitive diagnostic assessment (CDA) is the absence of tests developed for small-scale diagnostic purposes. Currently, much of the research carried out has been mainly on large-scale tests, e.g., TOEFL, MELAB, IELTS, etc. Even so, formative language assessment with a focus on informing instruction and engaging in identification of student’s strengths and weaknesses to guide instruction has not been conducted in the Iranian English language learning context. In an attempt to respond to the call for developing diagnostic tests, this study explored developing a cognitive diagnostic reading comprehension test for CDA purposes. To achieve this, initially, a list of reading attributes was prepared based on the literature and then the attributes were used to construct 20 reading comprehension items. Then seven content raters were asked to identify the attributes of each item of the test. To obtain quantitative data for Q-matrix construction, the test battery was administered to 1986 students of a General English Language Course at the University of Tehran, Iran. In addition, 13 students were recruited to participate in think-aloud verbal protocols. On the basis of the overall agreement of the content raters’ judgements concerning the choices of attributes and results of think-aloud verbal protocol analysis, a Q-matrix that specified the relationships between test items and target attributes was developed. Finally, to examine the CDA of the test, the Fusion Model, a type of cognitive diagnostic model (CDM), was used for diagnosing the participants' strengths and weaknesses. Results suggest that nine major reading attributes are involved in these reading comprehension test items. The results obtained from such cognitive diagnostic analyses could be beneficial for both teachers and curriculum developers to prepare instructional materials that target specific weaknesses and inform them of the more problematic areas to focus on in class in order to plan for better instruction.
Machine summary:
"Considering the issues of construct definition through item writing, and concluding with validation procedures rather than the CDM statistical models, the two approaches focus on the use of cognition in the process of item and test development (Leighton & Gierl, 2007).
According to Leighton and Gierl (2007), relying on existing test specifications for Q-matrices is usually unwarranted, so in this study an attempt was made at developing a reading comprehension test based on a cognitive diagnostic framework, with the help of Q-matrix construction.
Research Questions The purpose of this study is to develop a reading comprehension test based on a cognitive diagnostic framework and validate it by constructing a Q-matrix for the identification of underlying skills necessary to respond to the test items.
Attributes of L2 reading ability L2 Reading Attributes (View the image of this page) Procedure The study was carried out in three stages; 1) Developing the reading comprehension test based on a cognitive assessment framework; 2) Constructing and validating a Q-Matrix of reading attributes; 3) Statistical analysis of data.
The main goal of the current study was to develop a test based on a cognitive diagnostic framework in order to diagnose learners' strengths and weaknesses in L2 reading ability, with the ultimate goal of providing detailed information that can assist teachers and administrators for instructional purposes and to improve student performance.
While this study was an attempt at diagnostic assessment of L2 reading attributes, it also demonstrates a need for continued research in the area of cognitive diagnostic assessment in the Iranian context, particularly with regard to constructing diagnostic tests for different skills including writing, speaking and listening."