Browse > Article
http://dx.doi.org/10.5392/JKCA.2021.21.10.576

Verification of the Usefulness of the Mock TOEIC Test using Corpus Indices : Focusing on the Analysis of Difficulty and Discrimination  

Lee, Yena (차의과학대학교)
Publication Information
The Journal of the Korea Contents Association / v.21, no.10, 2021 , pp. 576-593 More about this Journal
Abstract
In this study, in order to investigate the factors that affect the percentage of correct answers and the degree of discrimination of the TOEIC test, a regression analysis was performed using corpus indicators that influence correct answer rate and the degree of discrimination for each part derived from the item analysis. The basic calculation word_length, consistency index LSA_overlap_adjacent_sentences, lexical diversity MTLD_VOCD, conjunction All_logical_causal_connectives_incidence, situational model casual_particles_causal_verbs_Ratio, syntactic complexity Left_embeddedness, and syntactic pattern density Infinitive_density were found to have negative effects. These factors that lower the correct answer rate can be utilized when setting learning goals. Vocabulary diversity index MTLD_VOCD, conjunction Additive_connectives_incidence, syntactic pattern density Infinitive_density, and lexical information person1_2_pronoun_incidence were found to have a positive effect. Factors influencing the increase in discrimination may provide important information for developing a learning program.
Keywords
Corpus; TOEIC; Usefulness; Item Analysis; Item Difficulty; Degree of Discrimination;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 J. P, Kincaid, R. P. Fishburne, R. L. Rogers, and B. S. Chissom, Derivation of new readability formulas (automated readability index, fog count, and flesch reading ease formula) for Navy enlisted personnel, Naval Technical Training Command Millington TN Research Branch, 1975.
2 A. C. Graesser, M. Jeon, Y. Yan, and Z. Cai, "Discourse cohesion in text and tutorial dialogue," Information Design Journal, Vol.15, No.3, pp.199-213, 2007.   DOI
3 A. C. Graesser, M. Jeon, Z. Cai, D. S. McNamara, J. Auracher, and W. van Peer, "Automatic analyses of language, discourse, and situation models," In J. Auracher and W. van Peer (Eds.), New beginnings in literary studies, UK: Cambridge Scholars Publishing, pp.82-88, 2008.
4 M. G. Jeon and I. J. Lim, "A corpus-based linguistic analysis of high school English textbooks," Foreign Languages Education, Vol.17, No.1, pp.209-233, 2010.
5 A. C. Graesser, D. S. McNamara, and J. M. Kulikowich, "Coh-Metrix: Providing multilevel analyses of text characteristics," Educational Researcher, Vol.40, No.5, pp.223-234, 2011.   DOI
6 M. Coltheart, "The MRC psycholinguistic database," The Quarterly Journal of Experimental Psychology A: Human Experimental Psychology, Vol.33, No.4, pp.497-505, 1981.   DOI
7 C. Fellbaum, WordNet: An electronic lexical database, Cambridge MA: MIT Press, 1998.
8 G. A. Miller, R. Beckwith, C. Fellbaum, D. Gross, and K. J. Miller, "Introduction to WordNet: An on-line lexical database," International Journal of Lexicography, Vol.3, No.4, pp.235-244, 1990.   DOI
9 N. J. Farr, J. J. Jenkins, and D. G. Paterson, "Simplification of Flesch Reading Ease Formula," Journal of Applied Psychology, Vol.35, No.5, pp.333-337, 1951.   DOI
10 T. S. Hock, "The role of prior knowledge and language proficiency as predictors of reading comprehension among undergraduates," In J. de Jong and D. Stevenson (Eds.), Individualizing the assessment of language abilities, England: Multilingual Matters, pp.214-244, 1990.
11 M. Jeon, Automatic analyses of cohesion and coherence in tutorial dialogue, Unpublished doctoral dissertation, University of Memphis, 2008.
12 J. S. Cangelosi, Designing tests for evaluating student achievement, New York: Longman, 1990.
13 B. Louw, "Irony in the text or insincerity in the writer? The diagnostic potential of semantic prosodies" In M. Baker and G. Francis, and E. T. Bonelli (Eds.), Text and technology: In honour of John Sinclair, PA: John Benjamins Publishing, pp.157-176, 1993.
14 W. Yang, X. Lu, and S. C. Weigle, "Different topics, different discourse: Relationships among writing topic, measures of syntactic complexity, and judgments of writing quality," Journal of Second Language Writing, Vol.28, pp.53-67, 2015.   DOI
15 H. J. Yoon and C. Polio, "The linguistic development of students of English as a second language in two written genres," T2ESOL Quarterly, Vol.51, No.2, pp.275-301, 2017.   DOI
16 J. Sinclair, "Corpus and text-basic principles," In M. Wynne (Ed.), Developing linguistic corpora: A guide to good practice, UK: Oxbow Books, pp.1-16, 2005.
17 R. L. Ebel, Measuring educational achievement, Englewood Cliffs, NJ: Prentice Hall, 1965.
18 N. Schmitt, Vocabulary in language teaching, UK: Cambridge University Press, 2000.
19 E. Charniak, A maximum-entropy-inspired parser, Association for Computational Linguistics conference (NACCL-1), 2000.
20 R. H. Baayen, R. Piepenbrock, and L. Gulikers, The CELEX lexical database (release 2), PA: Linguistic Data Consortium, 1995.
21 권인숙, "중학교 영어 교과서의 코퍼스 언어학적 어휘 비교 분석," 영어교육, 제57권, 제4호, pp.409-444, 2002.
22 J. C. Alderson, Assessing reading, UK: Cambridge University Press, 2000.
23 B. Davey and C. Lasasso, "The interaction of reader and task factors in the assessment of reading comprehension," The Journal of Experimental Education, Vol.52, No.4, pp.199-206, 1984.   DOI
24 X. Lu, "A corpus-based evaluation of syntactic complexity measures as indices of college-level ESL writers' language development," TESOL Quarterly, Vol.45, No.1, pp.36-62, 2011.   DOI
25 B. Bulte and A. Housen, "Conceptualizing and measuring short-term changes in L2 writing complexity," Journal of second language writing, Vol.26, pp.42-65, 2014.   DOI
26 X. Lu and H. Ai, "Syntactic complexity in college-level English writing: Differences among writers with diverse L1 backgrounds," Journal of Second Language Writing, Vol.29, pp.16-27, 2015.   DOI
27 우상희, 개정 7차 교육과정 고등영어 교과서 학습활동책의 수준별 난이도 분석, 고려대학교 교육대학원, 석사학위논문, 2009.
28 황이수, 이제영, "수학능력시험 영어 읽기 지문의 응집성과 문항 난이도 간의 상관관계 분석," 한국콘텐츠학회논문지, 제20권, 제5호, pp.344-350, 2020.   DOI
29 최민주, 김정렬, "수능 영어 문항 유형간 응집력과 어휘정보 분석," 한국콘텐츠학회논문지, 제17권, 제12호, 2017.
30 D. Biber, B. Douglas, S. Conrad, and R Reppen, Corpus linguistics: Investigating language structure and use, UK: Cambridge University Press, 1998.
31 M. Lewis, Teaching collocation: Further developments in the lexical approach, England: Language Teaching Publication, 2000.
32 J. Sinclair, Preliminary recommendations on corpus typology, UK: Corpus Linguistics Group, 1996.
33 D. Jurafsky and J. H. Martin, Speech and language processing: An introduction to speech recognition, computational linguistics and natural language processing, NJ: Prentice Hall, 2008.
34 E. Shohamy, "Does the testing method make a difference?," Language Testing, Vol.1, No.2, pp.147-170, 1984.   DOI
35 L. Ortega, "Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing," Applied Linguistics, Vol.24, No.4, pp.492-518, 2003.   DOI
36 N. Taguchi, W. Crawford, and D. Z. Wetzel, "What linguistic features are indicative of writing quality? A case of argumentative essays in a college composition program," TESOL Quarterly, Vol.47, No.2, pp.420-430, 2013.   DOI
37 S. Hong, "How can corpus data be applied for EFL learners and teachers?," English Language Teaching, Vol.22, No.2, pp.55-77, 2010.   DOI
38 이석재, 정채관, "한국 예비 대학생의 영어 사용 특성파악을 위한 대규모 공개 영어 학습자 코퍼스 구축 및 분석," 한국콘텐츠학회논문지, 제14권, 제11호, pp.1019-1029, 2014.   DOI
39 S. K. Shin, "The prediction of item difficulty in a listening test," Language Research, Vol.41, No.3, pp.683-692, 2005.
40 C. W. Liao, Y. Qu, and R. Morgan, "The relationships of test scores measured by the TOEIC listening and reading test and TOEIC speaking and writing tests," TOEIC Compendium Study, Vol.13, pp.1-15, 2010.
41 K. A. Jin and C. Park, "The prediction of English item difficulty in college scholastic ability test," English Teaching, Vol.59, No.1, pp.267-278, 2004.
42 D. D. Qian, "Investigating the relationship between vocabulary knowledge and academic reading performance: An assessment perspective," Language Learning, Vol.52, No.3, pp.513-536, 2002.   DOI
43 C. Clapham, The development of IELTS: A study of the effect of background on reading comprehension (Vol. 4), UK: Cambridge University Press, 1996.
44 Y. N. Lee, A validation study of TOEIC based on test usefulnes, Unpublished doctoral dissertation, Korea University, 2020.
45 M. Pearson and D. Johnson, Teaching reading comprehension, NJ: Holt, Rinehart and Winston, 1978.
46 J. C. Alderson and A. H. Urquhart, "The effect of students' academic discipline on their performance on ESP reading tests," Language Testing, Vol.2, No.2, pp.192-204, 1985.   DOI
47 N. Schmitt, "The relationship between TOEFL vocabulary items and meaning, association, collocation and word-class knowledge," Language Testing, Vol.16, No.2, pp.189-216, 1999.   DOI
48 P. Johnston, "Prior knowledge and reading comprehension test bias," Reading Research Quarterly, Vol.19, No.2, pp.219-239, 1984.   DOI
49 H. Ai and X. Lu, "A corpus-based comparison of syntactic complexity in NNS and NS university students' writing," In A. Diaz-Negrillo and N. Ballier, and P. Thompson (Eds.), Automatic treatment and analysis of learner corpus data, PA: John Benjamins Publishing, pp.249-264, 2013.