Journal of Gifted/Talented Education (영재교육연구)

The Korean Society for the Gifted (한국영재학회)
권호 표지
DOI QR코드

DOI QR Code

An Investigation into the Equivalence of Three Pictures for Creative Story Writing: 'Dog Owners', 'Lost Dog', and 'Overslept'

창의적 이야기 작문용 세 그림의 동형 조사: 'Dog Owners,' 'Lost Dog,' 'Overslept'

  • Received : 2016.11.16
  • Accepted : 2016.12.27
  • Published : 2016.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Alternate pictures that are proven to be equivalent are in high demand to assess creative thinking and language skills. This study aimed to investigate the equivalence of three pictures ('Dog owners,' 'Lost Dog,' and 'Overslept') recently developed for use in a creative writing task. Middle school students (N=183) wrote a story in English based on one of the three prompts distributed randomly. Four writing features (fluency, syntactic complexity, lexical diversity, and temporality) were analyzed with Coh-Metrix and MANCOVA. The three prompts were largely equivalent in their capacity to detect differences among writers in all the features of writing. The difficulty levels of the three prompts, however, were not necessarily the same. Two prompts, Dog Owners and Lost Dog, were verified as equivalent prompts, and therefore, they are recommended as alternate forms to assess creative language skills in repeated measurements. The Overslept prompt had greater facility in eliciting diverse words and more temporal connectives in composing stories. The differential difficulty shown among the prompts suggests that the validity of using different picture versions in repeated assessment remains questionable unless those versions undergo equivalence verification.

창의적 사고와 언어기술을 평가하는데 동형검사로 판명된 대체 그림들이 절실히 요구되고 있다. 본 연구는 창의적 쓰기 과제용으로 최근 개발된 세 그림(이름: 'Dog Owners,' 'Lost Dog,' 'Overslept')이 동형 검사지가 되는지 조사하였다. 183명의 중학생들이 무작위로 배분된 세 그림 중 하나에 의거하여 영어로 이야기를 작성하였다. 작문은 네 가지 쓰기요소(유창성, 어휘 다양성, 구조 복잡성, 그리고 시간성)에 대해 Coh-Metrix와 MANCOVA로 분석되었다. 이 세 그림은 변별력에 있어 대체로 위 모든 요소에 대해 비슷하였다. 그러나 이들의 난이도는 요소별로 볼 때 반드시 같지는 않았다. Dog Owners와 Lost Dog 그림은 변별력과 난이도에 있어 동형으로 판명되었다. 그러므로 이 두 그림은 반복 측정에서 타당한 동형 검사지로 추천된다. Overslept 그림은 다양한 어휘와 시간 연결사들을 유발시키는 데에 다른 두 그림 보다 용이하였다. 그림의 난이도가 다를 수 있다는 결과는 반복시험에서 대체 그림을 사용할 시 이들 그림이 동형 검정을 거치지 않고서는 그 타당성이 의심스러울 수 있음을 환기시켜 준다.

Keywords

References

  1. Ackerman, J. M. (1991). Reading, writing, and knowing: The role of disciplinary knowledge in comprehension and composing. Research in the Teaching of English, 25(2), 133-178.
  2. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: Authors.
  3. Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford, UK: Oxford University Press.
  4. Bae, J. (2007). Development of English skills need not suffer as a result of immersion: Grades 1 and 2 writing assessment in a Korean/English Two‐Way Immersion Program. Language Learning, 57(2), 299-332. https://doi.org/10.1111/j.1467-9922.2007.00410.x
  5. Bae, J. (2014). The confirmation of equivalence of two picture series for children's story writing: 'Pizza' and 'Amusement Park.' Journal of Educational Evaluation, 27(1), 209-229.
  6. Bae, J., & Lee, Y. S. (2011). The validation of parallel test forms: 'Mountain' and 'beach' picture series for assessment of language skills. Language Testing, 28(2), 155-177. https://doi.org/10.1177/0265532210382446
  7. Bae, J., Bentler, P. M., & Lee, Y. S. (2016). On the role of content in writing assessment. Language Assessment Quarterly, 13(4), 302-328. https://doi.org/10.1080/15434303.2016.1246552
  8. Bae, J., Jordahl, J., & Lee, Y. S. (2012, May). The effect of relative simplicity or complexity of picture prompt on language and creativity performances in writing. Paper presented at the 2012 Annual International Conference of the Korea English Language Testing Association, Seoul, Korea.
  9. Baker, E. L., & Quellmalz, E. (1979). Results of pilot studies: Effects of variations in writing task stimuli on the analysis of student writing performance. Studies in measurement and methodology. Work unit 1: Design and use of tests. Los Angeles: University of California at Los Angeles.
  10. Berman, R. A., & Slobin, D. I. (1994). Relating events in narrative: A crosslinguistic developmental study. Hillsdale, NJ: Erlbaum.
  11. Cleaver, B., Scheurer, P., & Shorey, M. (1993). Children's responses to silhouette illustrations in picture books. Rochester NY: International Visual Literacy Association.
  12. Cliffordson, C. (2004). Effects of practice and intellectual growth on performance on the Swedish Scholastic Aptitude Test (SweSAT). European Journal of Psychological Assessment, 20(3), 192-204. https://doi.org/10.1027/1015-5759.20.3.192
  13. Crisp, V., & Sweiry, E. (2006). Can a picture ruin a thousand words? The effects of visual resources in exam questions. Educational Research, 48(2), 139-154. https://doi.org/10.1080/00131880600732249
  14. Cronbach, L. J. (1947). Test 'reliability': Its meaning and determination. Psychometrika, 12(1), 1-16. https://doi.org/10.1007/BF02289289
  15. Cycowicz, Y. M., Friedman, D., Rothstein, M., & Snodgrass, J. G. (1997). Picture naming by young children: Norms for name agreement, familiarity, and visual complexity. Journal of Experimental Child Psychology, 65(2), 171-237. https://doi.org/10.1006/jecp.1996.2356
  16. DeCoster, J. (2006). Testing group difference using T-tests, ANOVA, and nonparametric measures. Retrieved June 30, 2016 from http://www.stat-help.com/notes.html.
  17. Department for Education and Employment (1998). The National Literacy Strategy: a framework for teaching. London: Author.
  18. Duff, K. (2012). Evidence-based indicators of neuropsychological change in the individual patient:Relevant concepts and methods. Archives of Clinical Neuropsychology, 27(3), 248-261. https://doi.org/10.1093/arclin/acr120
  19. Duran, N. D., McCarthy, P. M., Graesser, A. C., & McNamara, D. S. (2007). Using temporal cohesion to predict temporal coherence in narrative and expository texts. Behavior Research Methods, 39(2), 212-223. https://doi.org/10.3758/BF03193150
  20. Fillmore, C. (1982). Frame semantics. In The Linguistic Society of Korea (Ed.), Linguistics in the morning calm (pp. 111-137). Seoul: Hanshin.
  21. Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36(2), 193-202. https://doi.org/10.3758/BF03195564
  22. Greenhoot, A. D., & Semb, P. A. (2008). Do illustrations enhance preschoolers' memories for stories? Age-related change in the picture facilitation effect. Journal of Experimental Child Psychology, 99(4), 271-287. https://doi.org/10.1016/j.jecp.2007.06.005
  23. Halliday, M. A. K., & Hasan, R. (1976). Cohesion in English. London: Longman.
  24. Hausknecht, J. P., Halpert, J. A., Di Paolo, N. T., & Gerrard, M. O. (2007). Retesting in selection: A meta-analysis of coaching and practice effects for tests of cognitive ability. Journal of Applied Psychology, 92(2), 373-385. https://doi.org/10.1037/0021-9010.92.2.373
  25. Jung, J., & Bae, J. (2013). The influence of picture prompt variation on writing performance: 'Series' vs. 'Imagine Before and After.' English Language Teaching, 25(2), 27-46. https://doi.org/10.17936/pkelt.2013.25.2.002
  26. Kim, H., Koo, S., & Bae, J. (2015). Positive, negative, and nil effects of connectives in written stories:Analysis by proficiency groups. Linguistic Research, 32, 105-124. https://doi.org/10.17250/khisli.32..201507.006
  27. Kintsch, E. (2005). Comprehension theory as a guide for the design of thoughtful questions. Topics in Language Disorders, 25(1), 51-64. https://doi.org/10.1097/00011363-200501000-00006
  28. Koizumi, R., & In'nami, Y. (2012). Effects of text length on lexical diversity measures: Using short texts with less than 200 tokens. System, 40(4), 554-564. https://doi.org/10.1016/j.system.2012.10.012
  29. Laufer, B., & Nation, P. (1995). Vocabulary size and use: Lexical richness in L2 written production. Applied Linguistics, 16(3), 307-322. https://doi.org/10.1093/applin/16.3.307
  30. Lee, S-D., & Koh, W-J. (2013). The structural model analysis of related variables on English reading comprehension ability of gifted students. The Journal of the Korean Society for the Gifted and Talented, 12(1), 53-80.
  31. Lee, K., & Lee, Y. (2008). The effects of language activity programs using fairy tales on progress in children’s creativity. The Journal of the Koran Society for the Gifted and Talented, 7(1), 29-46.
  32. Liu, Y. (2002). Analyzing RM ANOVA related data using SPSS 10. Measurement in Physical and Exercise Science, 6(1), 43-60.
  33. Lomax, R. G. (2001). Statistical concepts: A second course for education and the behavioral sciences. Mahwah, NJ: Lawrence Erlbaum Publishers.
  34. McCaffrey, R. J., Duff, K., & Westervelt, H. J. (Eds.). (2000). Practitioner's guide to evaluating change with neuropsychological assessment instruments. New York: Kluwer Academic/Plenum Publishers.
  35. McCarthy, P. M., & Jarvis, S. (2007). vocd: A theoretical and empirical evaluation. Language Testing, 24(4), 459-488. https://doi.org/10.1177/0265532207080767
  36. McCarthy, P. M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior research methods, 42(2), 381-392. https://doi.org/10.3758/BRM.42.2.381
  37. McCutchen, D. (1986). Domain knowledge and linguistic knowledge in the development of writing ability. Journal of Memory and Language, 25(4), 431-444. https://doi.org/10.1016/0749-596X(86)90036-7
  38. McCutchen, D. (1996). A capacity theory of writing: Working memory in composition. Educational Psychology Review, 8(3), 299-325. https://doi.org/10.1007/BF01464076
  39. McCutchen, D. (2000). Knowledge, processing, and working memory: Implications for a Theory of Writing. Educational Psychologist, 35(1), 13-23. https://doi.org/10.1207/S15326985EP3501_3
  40. McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, N.J: Lawrence Erlbaum.
  41. McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2010). Linguistic features of writing quality. Written Communication, 27(1), 57-86. https://doi.org/10.1177/0741088309351547
  42. Meyers, L.S., Gamst, G., & Guarino, A. J. (2006). Applied multivariate research: design and interpretation. London: Sage Publications.
  43. Oswald, F. L., Friede, A. J., Schmitt, N., Kim, B. H., & Ramsay, L. J. (2005). Extending a practical method for developing alternate test forms using independent sets of items. Organizational Research Methods, 8(2), 149-164. https://doi.org/10.1177/1094428105275365
  44. Pearce, W. M. (2003). Does the choice of stimulus affect the complexity of children's oral narratives? Advances in Speech-Language Pathology, 5(2), 95-103. https://doi.org/10.1080/14417040510001669121
  45. Pena, E. D., Gillam, R. B., Malek, M., Ruiz-Felter, R., Resendiz, M., Fiestas, C., & Sabel, T. (2006). Dynamic assessment of school-age children's narrative ability: An experimental investigation of classification accuracy. Journal of Speech, Language, and Hearing Research, 49(5), 1037-1057. https://doi.org/10.1044/1092-4388(2006/074)
  46. Peterson, C., & Dodsworth, P. (1991). A longitudinal analysis of young children's cohesion and noun specification in narratives. Journal of Child Language, 18(2), 397-415. https://doi.org/10.1017/S0305000900011120
  47. Quereshi, M. Y. (2003). Absence of parallel forms for the traditional individual intelligence tests. Current Psychology, 22(2), 149-154. https://doi.org/10.1007/s12144-003-1005-7
  48. Raymond, M. R., Neustel, S., & Anderson, D. (2007). Retest effects on identical and parallel forms in certification and licensure testing. Personnel Psychology, 60(2), 367-396. https://doi.org/10.1111/j.1744-6570.2007.00077.x
  49. Schneider, P., & Dube, R. V. (2005). Story presentation effects on children's retell content. American Journal of Speech-Language Pathology, 14(1), 52-60. https://doi.org/10.1044/1058-0360(2005/007)
  50. Schweizer, M. L. (1999). The effect of content, style, and color of picture prompts on narrative writing: An analysis of fifth and Eighth grade students' writing. Unpublished doctoral dissertation, Virginia Polytechnic Institute and State University, Blacksburg, VA.
  51. Shapiro, L. R., & Hudson, J. A. (1991). Tell me a make-believe story: Coherence and cohesion in young children's picture-elicited narratives. Developmental Psychology, 27(6), 960-974. https://doi.org/10.1037/0012-1649.27.6.960
  52. Skehan, P. (1996). A framework for the implementation of task-based instruction. Applied Linguistics, 17(1), 38-62. https://doi.org/10.1093/applin/17.1.38
  53. Weir, C. J., & Wu, J. R. (2006). Establishing test form and individual task comparability: Case study of a semi-direct speaking test. Language Testing, 23(2), 167-197. https://doi.org/10.1191/0265532206lt326oa
  54. Wolfe-Quintero, K., Inagaki, S., & Kim, H. Y. (1998). Second language development in writing: Measures of fluency, accuracy and complexity. Honolulu, HI: University of Hawaii at Manoa.
  55. Young, K. M., & Leinhardt, G. (1998). Writing from primary documents: A way of knowing in history. Written Communication, 15(1), 25-68. https://doi.org/10.1177/0741088398015001002