Communications for Statistical Applications and Methods

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

A sample size calibration approach for the p-value problem in huge samples

  • Park, Yousung (Department of Statistics, Korea University) ;
  • Jeon, Saebom (Department of Marketing Information Consulting, Mokwon University) ;
  • Kwon, Tae Yeon (Department of International Finance, Hankuk University of Foreign Studies University)
  • Received : 2018.05.08
  • Accepted : 2018.06.25
  • Published : 2018.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

The inclusion of covariates in the model often affects not only the estimates of meaningful variables of interest but also its statistical significance. Such gap between statistical and subject-matter significance is a critical issue in huge sample studies. A popular huge sample study, the sample cohort data from Korean National Health Insurance Service, showed such gap of significance in the inference for the effect of obesity on cause of mortality, requiring careful consideration. In this regard, this paper proposes a sample size calibration method based on a Monte Carlo t (or z)-test approach without Monte Carlo simulation, and also proposes a test procedure for subject-matter significance using this calibration method in order to complement the deflated p-value in the huge sample size. Our calibration method shows no subject-matter significance of the obesity paradox regardless of race, sex, and age groups, unlike traditional statistical suggestions based on p-values.

Keywords

Acknowledgement

Supported by : National Research Foundation of Korea (NRF)

References

  1. Adams KF, Schatzkin A, Harris TB, Kipnis V, Mouw T, Ballard-Barbash R, Hollenbeck A, and Leitzmann MF (2006). Overweight, obesity, and mortality in a large prospective cohort of persons 50 to 71 years old, New England Journal of Medicine, 355, 763-778. https://doi.org/10.1056/NEJMoa055643
  2. Altman M (2004). Introduction to special issue on statistical significance, Journal of Socio-Economics, 33, 523-675. https://doi.org/10.1016/j.socec.2004.09.023
  3. Bayarri MJ and Berger JO (1999). Quantifying surprise in the data and model verification. In Bernardo JM, Berger JO, Dawid AP, and Smith AFM (eds) Bayesian Statistics (6th ed, pp. 53-82), Oxford University Press, Oxford.
  4. Bayarri MJ and Berger JO (2000). P-values for composite null models, Journal of the American Statistical Association, 95, 1127-1142.
  5. Berrington de Gonzalez A, Hartge P, Cerhan JR, et al. (2010). Body-mass index and mortality among 1.46 million white adults, New England Journal of Medicine, 363, 2211-2219. https://doi.org/10.1056/NEJMoa1000367
  6. Bolstad WM (2009). Understanding Computational Bayesian Statistics, John Wiley & Sons, Boston, MA.
  7. Carnethon MR, De Chavez PJ, Biggs ML, et al. (2012). Association of weight status with mortality in adults with incident diabetes, The Journal of American Medical Association, 308, 581-590.
  8. DeGroot MH and Schervish MJ (2002). Probability and Statistics (3rd ed), Pearson, Boston.
  9. Emerson S (2009). Small sample performance and calibration of the Empirical Likelihood method (Ph.D. thesis), Stanford University, Stanford.
  10. Flegal KM, Graubard BI, Williamson DF, and Gail MH (2007). Cause-specific excess deaths associated with underweight, overweight, and obesity, The Journal of Medical Association, 298, 2028-2037. https://doi.org/10.1001/jama.298.17.2028
  11. Fomby TB, Hill RC, and Jhonson SR (1984). Advanced Econometric Methods, Springer Verlag, New York.
  12. Gelman A, Meng XL, and Stern H (1996). Posterior predictive assessment of model fitness via realized discrepancies (with discussion), Statistica Sinica, 6, 733-807.
  13. Ghose A, Smith M, and Telang R (2006). Internet exchanges for used books: an empirical analysis of product cannibalization and welfare impact, Information Systems Research, 17, 3-19. https://doi.org/10.1287/isre.1050.0072
  14. Ghose A and Yang S (2009). An empirical analysis of search engine advertising: sponsored search in electronic markets, Management Science, 55, 1605-1622. https://doi.org/10.1287/mnsc.1090.1054
  15. Good IJ (1980). C73. The diminishing significance of a p-value as the sample size increase, Journal of Statistical Computation and Simulation, 11, 307-313. https://doi.org/10.1080/00949658008810416
  16. Good IJ (1982). C144. The diminishing significance of a fixed p-value as the sample size increase: a discrete model, Journal of Statistical Computation and Simulation, 16, 312-313.
  17. Greene WH (2003). Econometric Analysis (5th ed), Pearson Education, New Jersey.
  18. Halsey LG, Curran-Everett D, Vowler SL, and Drummond GB (2015). The fickle P value generate irreproducible results, Nature Methods, 12, 179-185. https://doi.org/10.1038/nmeth.3288
  19. Harlow LL, Mulaik S, and Steiger JH (2016). What If There were No Significance Tests?: Classic Edition (Chapter 2-5), Psychology Press, New York.
  20. Hubbard R and Armstrong JS (2006). Why we don't really know what statistical significance means: Implications for educators, Journal of Marketing Education, 28, 114-120. https://doi.org/10.1177/0273475306288399
  21. Johnson DH (1999). The insignificance of statistical significance testing, Journal of Statistical Computation and Simulation, 63, 763-772.
  22. Johnston J (1984). Econometric Methods (3rd ed), McGraw-Hill, New York.
  23. Kalantar-Zadeh K, Block G, Horwich T, and Fonarow GC (2004). Reverse epidemiology of conventional cardiovascular risk factors in patients with chronic heart failure, Journal of the American College of Cardiology, 43, 1439-1444. https://doi.org/10.1016/j.jacc.2003.11.039
  24. Kalantar-Zadeh K, Streja E, Molnar MZ, Lukowsky LR, Krishnan M, Kovesdy CP, and Greenland S (2012). Mortality prediction by surrogates of body composition: an examination of the obesity paradox in hemodialysis patients using composite ranking score analysis, American Journal of Epidemiology, 175, 793-803. https://doi.org/10.1093/aje/kwr384
  25. Katzmarzyk PT, Janssen I, and Ardern CI (2003). Physical inactivity, excess adiposity and premature mortality, Obesity Reviews, 4, 257-290. https://doi.org/10.1046/j.1467-789X.2003.00120.x
  26. Kim NH, Lee J, Kim TJ, et al. (2015). Body mass index and mortality in the general population and in subjects with chronic disease in Korea: a nationwide cohort study (2002-2010), PloS One, 10, e0139924. https://doi.org/10.1371/journal.pone.0139924
  27. Kirk RE (1996). Practical significance: a concept whose time has come, Educational and Psychological Measurement, 56, 746-759. https://doi.org/10.1177/0013164496056005002
  28. Lam BCC, Koh GCH, Chen C, Wong MTK, and Fallows SJ (2015). Comparison of Body Mass Index (BMI), Body Adiposity Index (BAI), Waist Circumference (WC), Waist-To-Hip Ratio (WHR) and Waist-To-Height Ratio (WHtR) as Predictors of Cardiovascular Disease Risk Factors in an Adult Population in Singapore, PLoS One, 10, e0122985. https://doi.org/10.1371/journal.pone.0122985
  29. Larsson SC and Wolk A (2008). Excess body fatness: an important cause of most cancers, The Lancet, 371, 536-537. https://doi.org/10.1016/S0140-6736(08)60247-0
  30. Leamer EE (1978). Specification Searches, John Wiley & Sons, New York.
  31. Lee J, Lee JS, Park SH, Shin SA, and Kim KW (2016). Cohort profile: the national health insurance service-national sample cohort (NHIS-NSC), South Korea, International Journal of Epidemiology, 46, e15-e15.
  32. Lin M, Lucas Jr HC, and Shmueli G (2013). Research commentary-too big to fail: large samples and the p-value problem, Information Systems Research, 24, 906-917. https://doi.org/10.1287/isre.2013.0480
  33. Nyamdorj R, Qiao Q, Lam TH, et al. (2008). BMI compared with central obesity indicators in relation to diabetes and hypertension in Asians, Obesity (Silver Spring), 16, 1622-1635. https://doi.org/10.1038/oby.2008.73
  34. Ogden CL, Carroll MD, Curtin LR, McDowell MA, Tabak CJ, and Flegal KM (2006). Prevalence of overweight and obesity in the United States, 1999-2004, The Journal of Medical Association, 295, 1549-1555. https://doi.org/10.1001/jama.295.13.1549
  35. Orpana HM, Berthelot JM, Kaplan MS, Feeny DH, McFarland B, and Ross NA (2010). BMI and mortality: results from a national longitudinal study of Canadian adults, Obesity, 18, 214-218. https://doi.org/10.1038/oby.2009.191
  36. Pawitan Y (2001). In All Likelihood: Statistical Modelling and Inference Using Likelihood, Oxford University Press, Oxford.
  37. Qian H and Shmidt P (2003). Partial GLS regression, Economic letters, 79, 385-392. https://doi.org/10.1016/S0165-1765(03)00033-8
  38. Renehan AG, Tyson M, Egger M, Heller RF, and Zwahlen M (2008). Body-mass index and incidence of cancer: a systematic review and meta-analysis of prospective observational studies, The Lancet, 371, 569-578. https://doi.org/10.1016/S0140-6736(08)60269-X
  39. Sellke T, Bayarri MJ, and Berger JO (2001). Calibration of p-values for testing precise null hypotheses, The American Statistician, 55, 63-71.
  40. Torloni MR, Betran AP, Horta BL, Nakamura MU, Atallah AN, Moron AF, and Valente O (2009). Prepregnancy BMI and the risk of gestational diabetes: a systematic review of the literature with meta-analysis, Obesity Reviews, 10, 194-203. https://doi.org/10.1111/j.1467-789X.2008.00541.x
  41. Tsao M (2001). A small sample calibration method for the empirical likelihood ratio, Statistics & Probability Letters, 54, 41-45. https://doi.org/10.1016/S0167-7152(01)00037-2
  42. Tsao M (2004). Bounds on coverage probabilities of the empirical likelihood ratio confidence regions, The Annals of Statistics, 32, 1215-1221. https://doi.org/10.1214/009053604000000337
  43. Uretsky S, Messerli FH, Bangalore S, Champion A, Cooper-Dehoff RM, Zhou Q, and Pepine CJ (2007). Obesity paradox in patients with hypertension and coronary artery disease, American Journal of Medicine, 120, 1863-1870.
  44. Wellek S (2017). A critical evaluation of the current p-value controversy, Biometrical Journal, 59, 854-872. https://doi.org/10.1002/bimj.201700001
  45. Whitehead J (1980). Fitting Cox's Regrssion Model to Survival Data using GLIM, Journal of the Royal Statistical Society. Series C (Applied Statistics), 29, 268-275.
  46. Zheng W, McLerran DF, Rolland B, Zhang X, Inoue M, Matsuo K, and Irie F (2011). Association between body-mass index and risk of death in more than 1 million Asians, New England Journal of Medicine, 364, 719-729. https://doi.org/10.1056/NEJMoa1010679