988368

1. Introduction
Handwriting as a visible form of spoken language [1] is an essential skill for children to develop in school activities [2, 3]. Failure to attain handwriting competency in school has far-reaching negative impacts on academic success, participation, and the self-efficacy of students [4]. Handwriting problems are observed in a significant number of cases referred to school-based occupational therapy [5]. Evaluation of handwriting function using standardized measures is required in both research and clinical settings [6, 7]. A number of handwriting evaluation tools are developed in different languages. However, they are inapplicable to evaluate Persian handwriting [8-10], because handwriting is a language dependent skill [11]. 
A few research studies have been performed to develop measures for assessing handwriting in Iran. However, they are weak and fail to assess students’ handwriting performance comprehensively [12-14]. Furthermore, most of the existing tools are limited to research on psychometric properties [12-15]. According to Feder and Majnemer, it is necessary to use comprehensive tools to evaluate handwriting performance [8]. Moreover, researchers or clinicians should ensure about the reliability and validity of a measuring instrument before drawing any conclusion or inferences from the collected data [16]. 
In the absence of comprehensive and standardized handwriting assessment tools in Iran, Havaei et al. (2016) developed the Persian Handwriting Assessment Tool (PHAT) for primary school-aged children (grades 2 and 3). The PHAT evaluates the legibility, speed of handwriting and orthographic mistakes in near-point copying and spelling. Ergonomic factors are also controlled in the evaluation process of this tool [17]. Developing a valid and reliable measure includes different procedures which are used sequentially at various stages of the measure construction [18]. 
Development processes, content validity and construct validity (discriminant validity and factor analysis) of the PHAT were reported previously. However, the reliability of PHAT is unclear [17]. This information is required before considering the PHAT as a useful tool for future clinical and research assessments. This study aimed to determine the reliability of the PHAT (i.e. internal consistency, & inter- and intra-rater reliability), in primary school children. 

2. Methods 
Study design and participants
Participants of this methodological study conducted during 2015-2016 were of grades 2 and 3 children (age range: 8 to 10 years). The participants were selected from 3 governmental primary schools by the random cluster sampling method out of 12 schools in the central region of Tehran City, Iran. Monolingual Persian speaking students with no history of neurological or bio-psychological impairments were included. Subjects were excluded if they had high stress and attention and concentration problems during test administration (7 students), and were bilingual students (14 students). A minimum sample size of 70 individuals was needed to examine the internal consistency [19].
Due to the effect of sample size on internal consistency results, this measure of the PHAT was investigated in 208 students (104 students from a girls’ school and 104 students from 2 boys’ schools). According to this method, a sample size of 30 to 50 is suitable to evaluate the intra- and inter-reliability [19]. Apart from samples used to survey the internal consistency, 48 students were recruited to evaluate the inter- and intra-rater reliability of PHAT. The samples were matched on gender and education. There was no missing value in the present study. 

Data collection
The PHAT evaluates the legibility, speed of handwriting and orthographic mistakes in primary school-aged children (grades 2 and 3). This tool was developed by Havaei et al. in 2016 and focuses on words readability in spelling and copying domains, as well as the speed of handwriting in copying domain. In addition, the orthographic mistakes are examined in spelling domain [17]. The legibility is evaluated considering the components of formation, alignment, space, size and text slant. Scoring of the legibility components are performed using a 5-point Likert-type Scale ranging from very poor to very good. The size of a word is scored from very small to very big. The subject score is the mean score of words in each component. The orthographic mistakes in spelling assignment are also recorded. Speed of handwriting is evaluated by 2 methods; 1. The required time to write copy assignment; and 2. The number of letters written per minute, calculated by a formula [17]. 
PHAT items in the copying and dictation domains are separately loaded into 3 components. This 3-D structure was designated and confirmed by exploratory and Confirmatory Factor Analysis (CFA and EFA), respectively. Content and discriminant validity of the PHAT is evident [17]. A silent and well-lighted room is needed for PHAT administration. The height of the table and chair should be appropriate for the participants. The required equipment include pencil (HB model), clipboard, eraser, sharpener, and a piece of printed lined A4 paper. An antislip writing aid should also be located on the table to prevent clipboard movement. The clipboard must be parallel to the forearm of the writing hand. This paper position enables the participant to see her/his writing assignment and to prevent smearing her/his writing [17]. 
In the current study, the participants were requested to copy 12 words (46 letters) from a near-point sample, while sitting on a desk in a silent room at school. The time spent to complete the task was recorded for each student to calculate speed of writing. Then, the student was asked to dictate 12 words (50 letters) on the paper. They were asked to write as “your usual good handwriting”. The PHAT was individually administrated on each subject by an occupational therapist. The PHAT is not a self-report tool, thus the rating of writing assignments was performed by pediatric occupational therapists with 12 years of working experience. The examiners received adequate training for rating children’s writing assignments.

Reliability
To determine the PHAT’s internal consistency, correlation between the words in writing assignments was investigated (each variable separately). For intra-rater reliability, the scoring of writing assignment was done by an occupational therapist in 2 weeks interval. For the evaluation of inter-rater reliability, scoring was performed by 2 pediatric occupational therapists under the same working experience and educational degree conditions.

Statistical analysis
Based on the Kolmogorov-Smirnov Test, the obtained data were normally distributed. The PHAT internal consistency was examined using the Cronbach alpha coefficient. Intraclass Correlation Coefficient (ICC, an index of relative reliability) analysis and Standard Error Measurement (SEM, an indicator of the absolute reliability) were used, to evaluate the intra- and inter-rater reliabilities. Data analysis was performed using SPSS [19].

3. Results
To determine the PHAT internal consistency, 208 students of grades 2 and 3 (104 boys, 104 girls; Mean±SD age: 8.99±0.653 years), participated in this study from 3 governmental primary schools. Also, 48 students (24 boys, 24 girls; Mean±SD age: 9.1±0.575 years) were recruited to examine the inter- and intra-rater reliabilities of PHAT. 
The PHAT internal consistency was determined using the Cronbach alpha. Results indicated good to excellent internal consistency (α=0.84-0.99) of the PHAT in dictation and copying domains (Table 1). The intra-rater reliability of PHAT as rated by an experienced occupational therapist with a 14-day interval varied from 0.87 to 1.00. The relevant results were reported through the ICC and SEM in Table 2. The degree of agreement among raters for inter-rater reliability of PHAT ranged from 0.7 to 1.00 (Table 3).

 

 

4. Discussion
The reliability is an important property of appropriate tools in research and clinical assessments. The internal and external reliability of tools must be assessed prior to use. For this purpose, we examined the internal consistency, inter-and intra-rater reliabilities of the PHAT. Results indicated good to excellent internal consistency for the measurements of PHAT in dictation and copying domains. 
A sample size of ˃50 is required for analyzing the internal consistency reliability. It also has an important role in the reliability coefficient. The large sample size of the current study (208 students) explains the favorable obtained results. Tseng (2013) reported moderate internal consistency (0.65) for measuring items of Chinese Handwriting Analysis System (CHAS). She discussed that handwriting is a complicated skill requiring cognitive, perceptual and sensory motor components. Thus, each item of a handwriting assessment tool may be influenced by different components, including planning, orthographic-motor integration, the character production of rapid movement sequences, self-monitoring, memory, and ideation [9, 20]. 
Rosenblum (2008) reported good internal consistency (0.9) for Handwriting Proficiency Screening Questionnaire. She mentioned that the bio-emotional status of participants, experienced rater, and large sample size influence the results [21]. Appropriate sampling according to the in clusion and exclusion criteria and rating the children’s assignment via expert occupational therapist may explain the good to excellent internal consistency in our study. Words selection (assignment) based on experts’ opinion and item analysis may have influenced the results.
 

 

 

 Our findings are significant when comparing the PHAT with other handwriting assessment tools. Most studies overlooked the internal consistency of tools and researchers relied on the expert opinion in the tool developing process [8].
Intra-rater and inter-rater reliability were assessed to examine the external reliability, because the PHAT is not a self-report tool. According to the results, the intra-rater reliability of the PHAT was good to excellent in copying and dictation domains (0.87 to 1). Duff (2010) reported an unacceptable intra-rater reliability for some variables of the Evaluation Tool of Children’s Handwriting (ETCH). He mentioned that long time for scoring, insufficient training of raters, unequal environmental conditions while scoring, and the complexity of handwriting assignment may explain the results. In Duff’s study, the lowest reliability related to writing from memory domain which is not recommended for handwriting assessment [22].
The low volume of the writing assignments of the PHAT reduces the scoring time. PHAT does not evaluate the handwriting components in writing from memory domain. It is also effective in reducing both administration and scoring time. Furthermore, scoring the assignments in the same context by a trained and expertise pediatric occupational therapist and selecting the appropriate words in writing assignment may have contributed in the present study findings. Joyce (2009) and Tseng (2013) reported that the working experience of raters is really important to evaluate students’ handwriting performance [9, 23]. Reisman (1993) also in her study about the development and validation of the Minnesota Handwriting Assessment (MHA) mentioned that the good results of the intra-rater reliability of MHT were due to the experience of therapist or rater and short scoring time [24]. 
Inter-rater reliability of the PHAT was lower than its intra-rater reliability (0.7 to 1). According to the literature, scoring handwriting legibility has a subjective nature [8, 22, 25]. Thus, the raters may consider different writing assignments. In our study, perhaps this has led to low scores compared with intra-rater reliability. Duff (2010) reported moderate inter-rater reliability for ETCH. Scoring ETCH is performed by total and analytic methods. Total scores suggested unacceptable reliability levels which is not a recommended tool [22]. Graham also mentioned that the total scoring method has a negative effect on the results of inter- and intra-rater reliabilities [26]. However, in the present study, scoring was performed by the analytic method (5-point Likert-type scale). This scoring method is more accurate than the total scoring method. Therefore, the inter-rater reliability of the PHAT was higher than the above-mentioned studies. Adequate training and experience of the rater also affect the results.
The subjective nature of scoring handwriting components may be the reason for the difference between the intra-rater and inter-rater reliability in the current study. In consistent with our study, Reisman (1993) in her study on the development of MHT and Feder (2003) in his review also, reported that intra-rater reliability was higher than inter-rater reliability [8, 24]. In the investigation of intra- and inter-rater reliability of the PHAT, reliability coefficients were 1 for speed (second and number) and orthographic mistake variables, because the rater did not interfere with scoring these variables [17]. The study limitations were as follows: the poor cooperation of some school administrators, time limitation in assessing participants, environmental noise, and the necessity of gender matching on the examiner and participants.
Appropriate sample size and scoring with analytic methods by expert raters and using writing assignment with confirmed words qualitatively (expert panel) and quantitatively (item analysis) are among the strong points of this study. This study could be more comprehensive if the scoring of teachers were compared with each other as well as therapists, to examine the inter-rater reliability of PHAT. Additionally, the internal consistency of the PHAT should be examined, when the teachers are raters. It is suggested to consider the test-retest reliability of PHAT, because personal and environmental characteristics affect handwriting performance.

5. Conclusion
PHAT is a reliable tool to evaluate handwriting components in primary school-aged children (grades 2 and 3). It can also be useful for Iranian therapists to identify elementary school students with handwriting problems.

Ethical Considerations
Compliance with ethical guidelines
The study protocol was approved by the Ethics Committee of Iran University of Medical Sciences (Code: IR.IUMS.REC.1394.9211525209). Informed consent was obtained from the parents or teachers of each participant. We ensured children that withdrawal from the study was voluntary. 

Funding
This study was financial supported by Iran University of Medical Sciences.

Authors contributions
The authors contributions is as follows: Conceptualization: Naser Havaei, Akram Azad, Mehdi Alizadeh-Zarei and Abbas Ebadi; Methodology: Naser Havaei, Akram Azad and Abbas Ebadi; Investigation: Naser Havaei, Akram Azad and Mehdi Alizadeh-Zarei Writing–original draft: Naser Havaei, Akram Azad and Abbas Ebadi; Writing–review & editing: Naser Havaei and Akram Azad; and Supervision: Akram Azad. 

Conflict of interest
The authors declared no conflict of interest.




References