Browse > Article
http://dx.doi.org/10.7468/jksmee.2014.28.4.431

Exploring the Application of Generalizability Theory to Mathematics Teacher Evaluation for Professional Development in Korea Based on the Analysis of Instructional Quality Assessment of Mathematics Teachers in the U.S.  

Kim, Sungyeun (Seoul National University)
Publication Information
Communications of Mathematical Education / v.28, no.4, 2014 , pp. 431-455 More about this Journal
Abstract
The purpose of this study was to suggest methods to apply generalizability theory to mathematics teacher evaluation using classroom observations in Korea by analysing mathematics teachers in the U.S. using the instructional quality of assessment instrument as an illustrative example. The subjects were 96 teachers participating in Year 3 and Year 4 from the Middle-school Mathematics and the Institutional Setting of Teaching (MIST) project funded by the National Science Foundation since 2007. The MIST project investigates the following question: What does it takes to support mathematics teachers' development of ambitious and equitable instructional practices on a large scale (MIST, 2007). This study examined data based on both the univariate generalizability analysis using GENOVA program and the multivariate generalizability analysis using mGENOVA program. Specifically, this study determined the relative effects of each error source and investigated optimal measuring conditions to obtain the suitable generalizability coefficients. The methodology applied in this study can be utilized to find effective optimal measurement conditions for the mathematics teacher evaluation for professional development in Korea. Finally, this study discussed limitations of the results and suggested directions for future research.
Keywords
mathematics instructional quality assessment; multivariate generalizability analysis; univariate generalizability analysis;
Citations & Related Records
Times Cited By KSCI : 4  (Citation Analysis)
연도 인용수 순위
1 Glaser-Zikuda, M., & FuB, S. (2008). Impact of teacher competencies on student emotions: A multi-method approach. International Journal of Educational Research, 47(2), 136-147.   DOI
2 Hattie, J. (2003). New Zealand education snapshot: with specific reference to the Yrs 1-13 years. Knowledge Wave Trust.
3 Hiebert, J., & Grouws, D. A. (2007). The effects of claBroom mathematics teaching on students' learning. Second Handbook of Research on Mathematics Teaching and Learning, 1, 371-404.
4 Ho, A. D., & Kane, T. J. (2013). The reliability of claBroom observations by school personnel. research paper. MET Project. Bill & Melinda Gates Foundation.
5 Hill, H. C., Blunk, M. L., Charalambous, C. Y., Lewis, J. M., Phelps, G. C., Sleep, L., & Ball, D. L. (2008). Mathematical knowledge for teaching and the mathematical quality of instruction: An exploratory study. Cognition and Instruction, 26(4), 430-511.   DOI
6 Hill, H. C., Charalambous, C. Y., & Kraft, M. A. (2012). When rater reliability is not enough: teacher observation systems and a case for the generalizability study. Educational Researcher, 41(2), 56-64.   DOI
7 Hill, H. C., Schilling, S. G., & Ball, D. L. (2004). Developing measures of teachers' mathematics knowledge for teaching. The Elementary School Journal, 105(1), 11-30.   DOI   ScienceOn
8 Jackson, K., Garrison, A., Wilson, J., Gibbons, L., & Shahan, E. (2013). Exploring relationships between setting up complex tasks and opportunities to learn in concluding whole-class discussions in middle-grades mathematics instruction. Journal for Research in Mathematics Education, 44(4), 646-682.   DOI
9 Jacobs, H. (2010). Race to the top. EMBO reports, 11(2), 73-73.   DOI
10 Kane, T. J., McCaffrey, D. F., Miller, T., & Staiger, D. O. (2013). Have we identified effective teachers? validating measures of effective teaching using random assignment. Research Paper. MET Project. Bill & Melinda Gates Foundation.
11 Kane, T. J., & Staiger, D. O. (2012). Gathering feedback for teachers: combining high-quality observations with student surveys and achievement gains. MET Project. Bill & Melinda Gates Foundation.
12 Kennedy, M. M. (2010). Attribution error and the quest for teacher quality. Educational Researcher, 39(8), 591-598.   DOI
13 Kyriakides, L., & Creemers, B. P. (2008). A longitudinal study on the stability over time of school and teacher effects on student outcomes. Oxford Review of Education, 34(5), 521-545.   DOI
14 Matsumura, L. C., Slater, S. C., Junker, B., Peterson, M., Boston, M., Steele, M., & Resnick, L. (2006). Measuring reading comprehension and mathematics instruction in urban middle schools: A pilot study of the instructional quality assessment. CSE Technical Report 681.
15 Learning Mathematics for Teaching Project. (2011). Measuring the mathematical quality of instruction. Journal of Mathematics Teacher Education, 14, 25-47.   DOI
16 Li, D., & Brennan, R. (2007). A multi-group generalizability analysis of a large-scale reading comprehension test. In annual meeting of the National Council on Measurement in Education. Chicago, IL.
17 Louisiana Act 54. (2010). Louisiana Departmemt of Education. Retrieved from http://www.louisianabelieves.com
18 Matsumura, L. C., Garnier, H. E., Slater, S. C., & Boston, M. D. (2008). Toward measuring instructional interactions "at-scale". Educational Assessment, 13(4), 267-300.   DOI
19 Middle-school Mathematics and the Institutional Setting of Teaching Project. (2007). MIST Project. Retrieved from http://peabody.vanderbilt.edu/departments/tl/teaching_and_learning_research/mist/index.php
20 Measures of Effective Teaching. (2009). Measures of effective teaching. Retrieved from www.metproject.org
21 Munter, C., & Correnti, R. (2011). Developing visions of high-quality mathematics instruction. Paper presented at the National Council of Teachers of Mathematics.
22 National Council of Teachers of Mathematics. (1991, 1993). Principles and standards for school mathematics. Reston, VA: Author.
23 National Center for Teacher Effectiveness. (2011). Online poll of states engaged in reform of teacher evaluation system. Cambridge, MA: Authors.
24 NuBbaum, A. (1984). Multivariate generalizability theory in educational measurement: An empirical study. Applied Psychological Measurement, 8(2), 219-230.   DOI
25 Nye, B., Konstantopoulos, S., & Hedges, L. V. (2004). How large are teacher effects?. Educational evaluation and policy analysis, 26(3), 237-257.   DOI
26 Shavelson, R. J., Baxter, G. P., & Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30(3), 215-232.   DOI   ScienceOn
27 Rockoff, J. E. (2004). The impact of individual teachers on student achievement: Evidence from panel data. American Economic Review, 94(2), 247-252.   DOI
28 Teddlie, C., & Reynolds, D. (Eds.). (2000). The international handbook of school effectiveness research. London: Falmer Press.
29 Rowan, B., Correnti, R., & Miller, R. (2002). What large-scale survey research tells us about teacher effects on student achievement: insights from the prospects study of elementary schools. The Teachers College Record, 104(8), 1525-1567.   DOI
30 Scheerens, J., & Bosker, R. J. (1997). The foundations of educational effectiveness. Oxford, UK: Pergamon.
31 Webb, N. M., Shavelson, R. J., & Maddahian, E. (1983). Multivariate generalizability theory. In L. J. Pyans, Jr.(Ed.), Genrealizability theory: Inferences and practical applications (pp67-81). San Francisco, CA:Jossey-Bass
32 Weisberg, D., Sexton, S., Mulhern, J., Keeling, D., Schunck, J., Palcisco, A., & Morgan, K. (2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness. New Teacher Project.
33 김성숙 (2001). 일반화가능도이론. 서울: 교육과학사.
34 김성숙 (1992). 관찰체계에 있어 측정의 변동요인 분석-관찰자 합치도, 안정도, 일반화가능도 비교. 교육평가연구, 5(1), 37-56.
35 김경선.이규민.강승혜 (2010). 일반화가능도 이론을 적용한 한국어 말하기 성취도 평가의 신뢰도와 오차요인 분석. 한국어 교육, 21(4), 51-75.
36 김도연.허종관 (2002). 일반화가능도이론을 적용한 주관적 배구기능검사의 신뢰도 추정. 한국체육측정평가학회지, 4(2), 15-28.
37 김보라.이규민 (2012). 일반화가능도 이론을 적용한 초등학교 쓰기 수행평가의 총체적 채점과 분석적 채점 방식 비교. 교육학연구, 50(4), 49-76.
38 김성숙 (1989a). 일반화가능도 이론을 이용한 교사행위 관찰에 있어서 오차원 분석. 교육평가연구, 3(1), 211-219.
39 김성숙 (2006). e-learning 강의평가 도구의 일반화가능도와 평가활용의 최적화 조건. 교육평가연구, 19(1), 305-322.
40 김성연 (2014a). 미국의 수업관찰평가 분석을 통한 우리나라 교원능력개발평가에서의 다변량 일반화가능도 이론 활용성 탐색. 한국교육, 41(1), 5-29.
41 김성연 (2014b). 미국 테네시 주 벤더빌트대학교 영재교육센터 프로그램이 우리나라 영재교육에 주는 시사점 탐색. 영재교육연구, 24(2), 217-243.
42 김성연.한기순 (2013). 관찰.추천제에 의한 수학영재 선발 시 사용되는 교사추천서와 자기소개서 평가에 대한 다변량 일반화가능도 이론의 활용. 영재교육연구, 23(5), 671-698.
43 김성연.한기순 (2014). 수학영재 선발에서 교사추천서와 자기소개서 채점내용 가중치에 따른 신뢰도 분석. 영재와 영재교육, 13(1), 43-65.
44 김성찬.김성연.한기순 (2012). 관찰.추천에 의한 수학영재 선발 시 사용되는 교사추천서와 자기소개서 평가에 대한 일반화가능도 이론의 활용. 한국수학교육학회 시리즈 E <수학교육 논문집>, 26(3), 251-271.
45 김현철 (2003). 일반화가능도 이론에 대한 대학평가의 신뢰도 추정과 효율적인 평가설계의 탐색. 교육학연구, 41(4), 49-70.
46 이규민.황경현 (2007). 초등학교 과학과 수행평가의 총체적 채점과 분석적 채점 방식에 대한 일반화가능도 분석. 아동교육, 16(4), 169-184.
47 교수학습개발센터 (2014). 교원전문성. 서울: 한국교육과정평가원.
48 이대현.최승현 (2006). 수학과 좋은 수업 사례에 대한 질적분석. 한국학교수학회논문집, 9(3), 249-263.
49 임형.김성숙 (2005). 임상수행능력평가의 오차요인 탐색과 신뢰도 연구. 교육평가연구, 18(1), 27-46.
50 이향 (2012). 말하기 수행 평가에서 발음 범주 채점의 최적화 방안 연구-일반화가능도 이론을 활용하여. 한국어교육, 23(2), 301-329.
51 교육과학기술부 (2013). 내실 있는 운영을 위한 2013년 교원능력개발평가제 개선 방안.
52 이현숙 (2012). 혼합형 검사의 문항 유형별 가중치에 따른 신뢰도 및 다변량 일반화가능도 분석. 교육평가연구, 25(1), 95-116.
53 조재윤 (2009). 일반화가능도 이론을 이용한 쓰기 평가의 오차원 분석 및 신뢰도 추정 연구. 국어교육, 128, 325-357.
54 최대현 (2014). 교원평가.근평.성과금 일원화 물밑작업-교육부, 67개 시범학교에서 3개 평가 지표 조정 적용키로. 교육희망, http://news.eduhope.net/sub_read.html?uid=15896
55 최승현.임찬빈 (2006). 수업평가 매뉴얼-수학과 수업평가 기준. 서울: 한국교육과정평가원.
56 최숙기 (2012). 국어과 수행 평가의 평가자 신뢰도 보고 방안 탐색-고등학생 요약문 평가 결과를 중심으로. 작문연구, 14, 395-424.
57 한혜정 (2012). 미국 뉴욕 주 교원임용 중등 교수역량 지필평가가 우리나라 교원임용시험에 주는 시사점 탐색. 한국교육, 39(2), 129-155
58 Boston, M. (2012). Assessing instructional quality in mathematics. The Elementary School Journal, 113(1), 76-104.   DOI
59 Brennan, R. L. (2001). Generalizability Theory. New York: Springer-Verlag.
60 Brennan, R. L. (1991). Manual for mGENOVA. A computer program for computing variance-covariance component. Iowa City, IA: Iowa Testing Program, University of Iowa.
61 Choi, Sungsook Kim. (1989b). An analysis of sources of variation in teacher behaviors using generalizability thoery. Doctoral dissertation. University of Virginia, Charlottesville, VA.
62 Cronbach, L. J., Gleser, G.C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: theory of generalizability for scores and profiles. New York: Wiley.
63 Cobb, P., & Jackson, K. (2011). Towards an empirically grounded theory of action for improving the quality of mathematics teaching at scale. Mathematics Teacher Education and Development, 13(1), 6-33.
64 Colorado Department of Education. (2012). 191 into action. Retrieved from http://www.coloradoea.org/
65 Cronbach, L. J., Linn, R. L., Brennan, R. L., & Haertel, E. H. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57(3), 373-399.   DOI
66 Danielson, C. (1996). Enhancing professional practice: A framework for teaching. Alexandria, Virginia: Association for Supervision and Curriculum Development.
67 Dunbar, S. B., Koretz, D. M., & Hoover, H. D. (1991). Quality control in the development and use of performance assessments. Applied measurement in education, 4(4), 289-303.   DOI
68 Eckert, J. M., & Dabrowski, J. (2010). Should value-added measures be used for performance pay?. Phi Delta Kappan, 91(8), 88-92.
69 임찬빈.이화진.곽영순.강대현.박영석 (2004). 수업평가 기준 개발 연구(1): 일반 기준 및 교과(사회, 과학, 영어). 서울: 한국교육과정평가원.
70 Crick, J. E., & Brennan, R. L. (1983). Manual for GENOVA: A GENeralized Analysis Of VAriance System, ACT Technical Bulletin, 43, The American College Testing Program.
71 Tennessee Department of Education. (2014). Tennessee Educator Acceleration Model. Retrieved from team-tn.org