• Journal of Korean language education
• /
• v.28 no.1
• /
• pp.173-196
• /
• 2017

Writing is a subjective and performative activity. Writing ability has multi-facets and compoundness. To understand the examinees's writing ability accurately and provide effective writing scores, raters first ought to have the competency regarding assessment. Therefore, this study is significant as a fundamental research about rater's characteristics on the TOPIK writing assessment. 150 scripts of the 47th TOPIK examinees were selected randomly, and were further rated independently by 20 raters. The many-facet Rasch model was used to generate individualized feedback reports on each rater's relative severity and consistency with respect to particular categories of the rating scale. This study was analyzed using the FACETS ver 3.71.4 program. Overfit and misfit raters showed many difficulties for noticing the difference between assessment factors and interpreting the criteria. Writing raters appear to have much confusion when interpreting the assessment criteria, and especially, overfit and misfit teachers interpret the criteria arbitrarily. The main reason of overfit and misfit is the confusion about assessment factors and criteria in finding basis for scoring. Therefore, there needs to be more training and research is needed for raters based on this type of writing assessment characteristics. This study is recognized significantly in that it collectively examined writing assessment characteristics of writing raters, and visually confirmed the assessment error aspects of writing assessment.

• Research in Mathematical Education
• /
• v.10 no.1 s.25
• /
• pp.55-70
• /
• 2006

The purpose of this study was to develop math creative problem solving test in order to identify the mathematically gifted on the basis of their math creative problem solving ability and evaluate the goodness of the test in terms of its reliability and validity of measuring creativity in math problem solving on the basis of fluency in producing valid solutions. Ten open math problems were developed requiring math thinking abilities such as intuitive insight, organization of information, inductive and deductive reasoning, generalization and application, and reflective thinking. The 10 open math test items were administered to 2,029 Grade 5 students who were recommended by their teachers as candidates for gifted education programs. Fluency, the number of valid solutions, in each problem was scored by math teachers. Their responses were analyzed by BIGSTEPTS based on Rasch's 1-parameter item-response model. The item analyses revealed that the problems were good in reliability, validity, difficulty, and discrimination power even when creativity was scored with the single criteria of fluency. This also confirmed that the open problems which are less-defined, less-structured and non-entrenched were good in measuring math creativity of the candidates for math gifted education programs. In addition, it discriminated applicants for two different gifted educational institutions and between male and female students as well.

• English Language & Literature Teaching
• /
• v.12 no.1
• /
• pp.221-238
• /
• 2006

This study investigates the reliability of plagiarism measurement in the ESL essay writing context. The current study aims to address the answers to the following research questions: 1) How does plagiarism measurement affect test reliability in a psychometric view? and 2) how do raters conceive the plagiarism in their analytic scoring? This study uses the mixed-methodology that crosses quantitative-qualitative techniques. Thirty eight international students took an ESL placement writing test offered by the University of Illinois. Two native expert raters rated students' essays in terms of 5 analytic features (organization, content, language use, source use, plagiarism) and made a holistic score using a scoring benchmark. For research question 1, the current study, using G-theory and Multi-facet Rasch model, found that plagiarism measurement threatened test reliability. For research question 2, two native raters and one non-native rater in their email correspondences responded that plagiarism was not a valid analytic area to be measured in a large-scale writing test. They viewed the plagiarism as a difficult measurement are. In conclusion, this study proposes that a systematic training program for avoiding plagiarism should be given to students. In addition, this study suggested that plagiarism is measured reliably in the small-scale classroom test.

• Journal of Korean Academy of Nursing
• /
• v.46 no.1
• /
• pp.109-117
• /
• 2016

Purpose: The aim of this study was to develop a Korean version of Night Eating Questionnaire (KNEQ) and test its psychometric properties and evaluate items according to item response theory. Methods: The 14-item NEQ as a measure of severity of the night eating syndrome was translated into Korean, and then this KNEQ was evaluated. A total of 1171 participants aged 20 to 50 completed the KNEQ on the Internet. To test reliability and validity, Cronbach's alpha, correlation, simple regression, and factor analysis were used. Each item was analyzed according to Rasch-Andrich rating scale model and item difficulty, discrimination, infit/outfit, and point measure correlation were evaluated. Results: Construct validity was evident. Cronbach's alpha was .78. The items of evening hyperphagia and nocturnal ingestion showed high ability in discriminating people with night eating syndrome, while items of morning anorexia and mood/sleep provided relatively little information. The results of item analysis showed that item2 and item7 needed to be revised to improve the reliability of KNEQ. Conclusion: KNEQ is an appropriate instrument to measure severity of night eating syndrome with good validity and reliability. However, further studies are needed to find cut-off scores to screen persons with night eating syndrome.

• Anxiety and mood
• /
• v.13 no.2
• /
• pp.115-122
• /
• 2017

Objective : The experiences in close relationships questionnaire-revised (ECR-R) (Fraley, Waller & Brennan, 2000) is a valuable tool for measuring adult attachment, and its Korean version, the ECRR-K (Kim, 2004), is widely used in Korea. However, given its substantial length, this study was aimed to develop and validate a short version of the ECRR-K called the ECRR-K14. Methods : Two hundred and ninety-four medical students participated in this study in 2016. They completed the ECRR-K, the Perceived Stress Scale (PSS), the Rosenberg Self-Esteem Scale (RSES), and the UCLA Loneliness Scale (UCLA-LS). The study authors applied the Rasch rating scale to check each item's model fit and then performed confirmatory factor analyses (CFAs) to test the new scale's validity. Results : The authors selected seven items each for the anxiety and avoidance subscales, and the ECRR-K14 showed fair to good internal consistency (Cronbach's ${\alpha}=0.93$ and 0.92 for anxiety and avoidance, respectively). The anxiety subscale showed concurrent validity with the PSS and the RSES while the avoidance subscale showed concurrent validity with the UCLA-LS. The CFAs also demonstrated the validity of the model with a goodness-of-fit index of 0.916. Conclusion : The ECRR-K14 showed excellent reliability and validity and appears to be a promising instrument for measuring the two attachment dimensions in adults.

• Physical Therapy Korea
• /
• v.19 no.3
• /
• pp.91-97
• /
• 2012

The primary aim of this study was to compare responsiveness of self-report by worker and therapist-scored functional capacity instrument. Self-report and therapist-scored interval-level person measures and item difficulties were compared at admission and discharge. Therapist and worker ratings were collected on 230 clients from 27 rehabilitation sites using the newly developed Occupational Rehabilitation Data Base (ORDB) functional capacity instrument. ORDB comprises several subscales measuring relevant variables of "a return-to-work model" in work-related rehabilitation clinics. The functional capacity scale deals with 10 DOT job factors. The rating scale categories were 1-severely impaired, 2-moderately impaired, 3-mildly impaired, and 4-not impaired. Only data from clients with low back pain (n=98) with complete data (both admission and discharge scores) were used for the present study. Therapists and workers completed the functional capacity instrument at admission and discharge. Rasch analysis [1-parameter item response theory model (IRT)] was applied to calibrate item difficulty and person ability measure of therapist and workers ratings. Effect sizes for therapist and self-report ratings were slightly different, .69 and .30, respectively. Therapist and worker ratings were more consistent at discharge (r=.54) than at admission (r=.32). Workers have a tendency to be more severe in their ratings (show higher item difficulties) than therapists at admission and discharge. Therapists and workers report similar magnitudes of improvement following treatment program. These findings challenge the belief that injured workers may unreliable source for monitoring therapeutic outcomes. Self-report measures have the advantage of conserving therapist time for treatment (versus evaluation). While the therapist and self-report ratings are comparable at discharge, there is less consistency at admission. Comparable therapist-worker ratings may be achieved by controlling for rating severity using IRT methodologies.

• The Journal of Korean Academy of Sensory Integration
• /
• v.21 no.2
• /
• pp.45-57
• /
• 2023

Objective : The purpose of this study was to investigate the concurrent validity of the self- and proxy-report versions of the KIDSCREEN-10 quality of life questionnaire. Methods : A total of nine children and nine parents were selected to represent a cohort registered for a school-based wellness program. Two versions of the KIDSCREEN-10 questionnaire (self- and proxy reports) were administered to the children and their parents. The Rasch rating scale model was applied to determine the dimensionality and item difficulty of the two versions of the questionnaire. Moreover, the item-person matching map and Spearman's rho were compared to confirm the concurrent validity of the two versions. Results : All items, except four items (i.e., autonomy, home life, concentration/learning, and peers/social support), fit the Rasch rating scale model of the children's self-report version of the questionnaire. With regard to the parent's proxy-report version, two items misfit the model. While the items of the self- and proxy-report versions showed similar item difficulties, the parents had a tendency to be more severe in their ratings than the children. The correlation between the two versions was relatively low (Spearman's rho = .533, p > .05). The scatterplots between the two versions showed differences in the item difficulties of the physical and psychological well-being and self-perception items. Conclusion : These findings suggest that the three identified items should be taken into consideration when measuring children's health-related quality of life using the KIDSCREEN-10 questionnaire.

• Korean Journal of Community Nutrition
• /
• v.16 no.6
• /
• pp.771-781
• /
• 2011

The objective of this study was to assess the reliability and validity of food security measures, which was developed based on the US household food security survey module (US HFSSM) with content validity in the Korean population. The reliability and validity were assessed by internal consistency, construct validity and criterion-related validity. The study included 446 households. Among those, 46.2% were households with children. The proportion of food insecure households was 33.3%. Among those, 35.4% and 64.6% households were food insecure with hunger and without hunger, respectively. The Cronbach's alpha coefficients were 0.84 and the infit value by the Rasch model analysis ranged from 0.68 to 1.43. The scale item response curves by food insecurity severity explained well the nature and characteristics of food security, indicating the highest proportion of "yes" for the items on diet quality, followed by those with diet quantity. The result of criterion-related validity showed that food insecurity status was significantly related in a dose-response manner with the household income level, food expenditure, subjective health state, subjects' educational level. Household food security status was also related to dietary diversity regarding protein foods, fruits and fruit juice, and milk and dairy product. These findings suggest that the food security instrument is reliable and valid and would be used to assess food security status in the Korean population.

• Journal of Korean Academy of Nursing
• /
• v.42 no.5
• /
• pp.632-643
• /
• 2012

Purpose: This study was done to develop tool to evaluate the core competencies regarding nursing cancer patients on chemotherapy, and to verify the reliability and efficacy of the developed tool. Methods: A tool to evaluate the core competencies was developed from a preliminary tool consisting of 112 items verified by expert groups. The adequacy of the preliminary tool was analyzed and refined to the final evaluation tool containing 76 items in 8 core competencies and 18 specific competencies. The evaluation tool is in the form of a self-report, and each item is evaluated according to a 3-point scale. From September 22 to October 14, 2011, 349 survey responses were analyzed using SPSS 20.0 and the WINSTEPS program that employs the Rasch model. Results: Results indicated that there were no inappropriate items and the items had low levels of difficulty in comparison with the knowledge levels of the study participants. The results of factor analysis yielded 18 factors, and the reliability of the tools was very high with Cronbach's ${\alpha}$=.97. Conclusion: The results of this study can be used for training and evaluation of core competencies for nursing cancer patients, and for standardizing nursing practices associated with chemotherapy.

• Journal of the Korean School Mathematics Society
• /
• v.9 no.3
• /
• pp.405-424
• /
• 2006

The purpose of this study was to develop the instruments which can measure behavior characteristics as a component of Mathematically Giftedness with in middle school period. This study prescribed the variable factors of measurement after classify the characteristics of Mathematically Giftedness through literature studies. And it produced instruments those are finally composed of 51 items through the preliminary test. The participants for the study were 424 Korean middle school students. Statistical analyses were carried out to verify the validities and reliability. Reliability(Cronbach $\alpha$) was in behavior characteristics, .95. Content validity was found to be satisfactory by internal validity evaluation on the test items. Internal validity were analyzed by BIGSTEPTS based on Rasch's 1-parameter item-response model. Construct validity was also found to be satisfactory through factor analysis which showed the four factors which the identification instruments were intended to measure such as, General mathematical mental ability, Mathematical Ability, Processing and Obtaining mathematical information Anility and Mathematical Disposition Ability. In conclusion, the instruments about behavior characteristics of Mathematically Giftedness during middle school period developed by this study are highly reliable on its reliability and validity.