1.3 Overview of research design

The objective of the study is to explore the relationship between writing proficiency of pre-service EFL teachers and their judgments of student writing.In addressing the major research questions,a mixed-methods design has been adopted with two related empirical studies.

1.3.1 Study One:Writing proficiency of pre-service EFL teachers

Study One was designed to examine differences of writing proficiency of pre-service EFL teachers characterized by discourse features inherent in their writing performances.The discourse analytic approach to writing performances was adopted to provide empirical evidences underlying expert judgments of teachers'writing proficiency levels.Nineteen discourse features were identified based on a systematic document analysis on standards/requirements on writing proficiency of EFL teachers in major English-speaking countries and non-English-speaking regions and countries.Measures that have been used in a range of previous studies and produced reliable,meaningful results,and that have clear theoretical justifications were chosen to operationalize the selected features.Eighty-one pre-service EFL teachers enrolled in one language education program in one key normal university in Mainland China participate in the study.A writing test was designed to elicit their actual writing performances,which were later assessed by language testing experts into three proficiency levels.

Analysis of Variance(ANOVA)was employed and eleven out of nineteen features covering all discourse categories investigated were able to successfully differentiate between writing performances of pre-service EFL teachers at three assessed levels.In general,the findings of Study One were in line with previous empirical L2 writing studies,providing a detailed picture of writing performances of pre-service EFL teachers.

1.3.2 Study Two:Scoring judgments of pre-service EFL teachers

To address Research Question 2,Study Two investigated the scoring processes of pre-service EFL teachers to determine how their judgments teachers related to their levels of writing proficiency.Think-aloud protocols were employed to detect variability in scoring judgments of teachers at three levels of writing proficiency.Based on the results of Study One,twelve pre-service EFL teachers at three levels of writing proficiency were identified.The student scripts under real examination conditions were collected from one intact class of one key senior high school.The twelve participants were required to take approximately fifty minutes to rate twelve selected student scripts while at the same time conducting concurrent thinking-aloud protocols to report their thought processes during rating.The interview data were also collected to facilitate the interpretation of qualitative data.

Data analysis comprised two main stages.Firstly,MFRM was used to analyze the scores awarded by the participants.The transcribed think-aloud protocols were coded and frequency data of each category of the coding schemes were analyzed quantitatively as the basis for comparison across teachers at three levels of writing proficiency.Second,interview data were analyzed thematically to identify significant dimensions of differences or similarities across the three groups.The quantitative picture of scoring judgments of pre-service EFL teachers was further combined with qualitative details regarding their scoring processes.Table 1-1 summarizes the research questions,data collection and analysis methods of the two studies.