Restorative Dentistry and Endodontics

The Korean Academy of Conservative Dentistry (대한치과보존학회)
권호 표지
DOI QR코드

DOI QR Code

Sample size determination for conducting a pilot study to assess reliability of a questionnaire

  • Mohamad Adam Bujang (Clinical Research Center, Sarawak General Hospital, Ministry of Health) ;
  • Evi Diana Omar (Sector of Biostatistics and Data Repository, National Institutes of Health) ;
  • Diana Hui Ping Foo (Clinical Research Center, Sarawak General Hospital, Ministry of Health) ;
  • Yoon Khee Hon (Institute for Clinical Research, National Institutes of Health, Ministry of Health Malaysia)
  • Received : 2023.09.27
  • Accepted : 2023.10.30
  • Published : 2024.02.28
Download PDF
⟨ Previous Next ⟩

Abstract

This article is a narrative review that discusses the recommended sample size requirements to design a pilot study to assess the reliability of a questionnaire. A list of various sample size tables that are based on the kappa agreement test, intra-class correlation test and Cronbach's alpha test has been compiled together. For all calculations, type I error (alpha) was set at a maximum value of 0.05, and power was set at a minimum value of 80.0%. For the kappa agreement test, intra-class correlation test, and Cronbach's alpha test, the recommended minimum sample size requirement based on the ideal effect sizes shall be at least 15, 22, and 24 subjects respectively. By making allowances for a non-response rate of 20.0%, a minimum sample size of 30 respondents will be sufficient to assess the reliability of the questionnaire. The clear guideline of minimum sample size requirement for the pilot study to assess the reliability of a questionnaire is discussed and this will ease researchers in preparation for the pilot study. This study provides justification for a minimum requirement of a sample size of 30 respondents specifically to test the reliability of a questionnaire.

Keywords

References

  1. Moore CG, Carter RE, Nietert PJ, Stewart PW. Recommendations for planning pilot studies in clinical and translational research. Clin Transl Sci 2011;4:332-337.
  2. Jalaludin M, Fuziah M, Hong J, Mohamad Adam B, Jamaiyah H. Reliability and validity of the revised Summary of Diabetes Self-Care Activities (SDSCA) for Malaysian children and adolescents. Malays Fam Physician 2012;7:10-20. 
  3. Yunus A, Seet W, Mohamad Adam B, Haniff J. Validation of the Malay version of Berlin questionaire to identify Malaysian patients for obstructive sleep apnea. Malays Fam Physician 2013;8:5-11.
  4. Cohen JA. Coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:37-46. 
  5. Feldman RS, Douglass CW, Loftus ER, Kapur KK, Chauncey HH. Interexaminer agreement in the measurement of periodontal disease. J Periodontal Res 1982;17:80-89.
  6. Lemasney J, O'Mullane D, Coleman M. Effect of fluoridation on dental health in 5- and 11-year-old Irish schoolchildren. Community Dent Oral Epidemiol 1984;12:218-222.
  7. Mann J, Goultschin J, Call RL. Assessment of inter-examiner agreement in scoring periodontal disease. J Periodontal Res 1985;20:86-90.
  8. Bujang MA, Baharum N. A simplified guide to determination of sample size requirements for estimating the value of intraclass correlation coefficient: a review. Arch Orofac Sci 2017;12:1-11. 
  9. Bartko JJ. On various intraclass correlation reliability coefficients. Psychol Bull 1976;83:762-765.
  10. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika 1951;16:297-334.
  11. Leontitsis A, Pagge J. A simulation approach on Cronbach's alpha statistical significance. Math Comput Simul 2007;73:336-340.
  12. Flack VF, Afifi AA, Lachenbruch PA, Schouten HJA. Sample size determinations for the two rater kappa statistic. Psychometrika 1988;53:321-325.
  13. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-174.
  14. Fleiss JL. Statistical methods for rates and proportions. 2nd ed. New York, NY: John Wiley; 1981. 
  15. Bujang MA, Baharum N. Guidelines of the minimum sample size requirements for Kappa agreement test. Epidemiol Biostat Public Health 2017;14:e12267-1-e12267-10.
  16. Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med 1998;17:101-110.
  17. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol 1994;6:284-290.
  18. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 2016;15:155-163.
  19. Bonett DG. Sample size requirements for testing and estimating coefficient alpha. J Educ Behav Stat 2002;27:335-340.
  20. Nunnally JC. Psychometric theory. New York, NY: McGraw-Hill; 1967. 
  21. Bujang MA, Omar ED, Baharum NA. A review on sample size determination for Cronbach's alpha test: a simple guide for researchers. Malays J Med Sci 2018;25:85-99.
  22. Hertzog MA. Considerations in determining sample size for pilot studies. Res Nurs Health 2008;31:180-191.
  23. Johanson GA, Brooks G. Initial scale development: sample size for pilot studies. Educ Psychol Meas 2010;70:394-400.
  24. Bujang MA. A step-by-step process on sample size determination for medical research. Malays J Med Sci 2021;28:15-27.
  25. Bujang MA, Ab-Ghani P, Soelar SA, Zulkifli NA, editors. Sample size guideline for exploratory factor analysis when using small sample: taking into considerations of different measurement scales. 2012 International Conference on Statistics in Science, Business and Engineering (ICSSBE); 2012 Sep 10-12; Langkawi, Malaysia. Piscataway, NJ: IEEE; 2012 Dec. p. 1-5. 
  26. Bujang MA, Adnan TH. Requirements for minimum sample size for sensitivity and specificity analysis. J Clin Diagn Res 2016;10:YE01-YE06.
  27. Bujang MA, Ghani PA, Soelar SA, Zulkifli NA, Omar ED. Invalid skewed responses contributes to invalid factor solution in exploratory factor analysis: a validation approach using real-life data. J Behav Health 2019;8:152-160.
  28. Bujang MA. An elaboration on sample size planning for performing a one-sample sensitivity and specificity analysis by basing on calculations on a specified 95% confidence interval width. Diagnostics (Basel) 2023;13:1390.
  29. Bujang MA, Mastura I, Bariyyah K, Baharum N, Othman SH, Mat-Lazim SS, et al. Validation of the summary of diabetes self-care activities (SDSCA) in Malay language for Malaysian adults. Malays J Public Health Med 2016;16:227-234. 
  30. Bujang MA, Lai WH, Ratnasingam S, Tiong XT, Hon YK, Yap EP, et al. Development of a quality-of-life instrument to measure current health outcomes: health-related quality of life with six domains (HRQ-6D). J Clin Med 2023;12:2816.