Browse > Article
http://dx.doi.org/10.5351/KJAS.2016.29.6.999

Current trends in high dimensional massive data analysis  

Jang, Woncheol (Department of Statistics, Seoul National University)
Kim, Gwangsu (Data Science for Knowledge Creation Research Center, Seoul National University)
Kim, Joungyoun (Department of Information Statistics, Chungbuk National University)
Publication Information
The Korean Journal of Applied Statistics / v.29, no.6, 2016 , pp. 999-1005 More about this Journal
Abstract
The advent of big data brings the opportunity to answer many open scientic questions but also presents some interesting challenges. Main features of contemporary datasets are the high dimensionality and massive sample size. In this paper, we give an overview of major challenges caused by these two features: (1) noise accumulation and spurious correlations in high dimensional data; (ii) computational scalability for massive data. We also provide applications of big data in various fields including forecast of disasters, digital humanities and sabermetrics.
Keywords
big data; computation scalability; digital humanities; noise accumulation; spurious correlations;
Citations & Related Records
Times Cited By KSCI : 3  (Citation Analysis)
연도 인용수 순위
1 Lee, B., Kim, D., Kim, D., and Jeong, H. (2016). N-gram web service and stylometric analysis of Korean historical documents. New Physics: Sae Mulli, 66, 502-510.   DOI
2 Park, J.-H., Lee, S.-Y., Kang, D. H. and Won, J.-H. (2013). Hadoop and Mapreduce. Journal of the Korean Data & Information Science Society, 24, 1013-1027.   DOI
3 Sawchik, T. (2015). Big Data Baseball: Math, Miracles, and the End of a 20 Year Losing Streak, Flatiron Books.
4 Sievert, C. (2015). Tools for harnessing 'MLBAM', 'Gameday' data and visualizing 'pictchfx', http://cpsievert.github.com/pitchRx
5 Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B, 58, 267-288.
6 van de Geer, S. Buhlmann, P. Ritov, Y. A., and Dezeure, R. (2014). On asymptotically optimal confidence regions and test for high-dimensional models. Annals of Statistics, 42, 1166-1202.   DOI
7 Witten, R. and Candes, E. (2013). Randomized algorithms for low-rank matrix factorizations: sharp performance bounds. Algorithmica, 63, 355-363.
8 Zhang, C.-H. (2010.) Nearly unbiased variable selection under minimax concave penalty. Annals of Statistics, 38, 894-942.   DOI
9 Clauset, A., Shalizi, C. R., and Newman, M. E. J. (2009). Power-law distributions in empirical data. SIAM Review, 51, 661-703.   DOI
10 Fan, J., Han, F. and Liu, H. (2014). Challenges of big data analysis. National Science Review, 1, 293-314.   DOI
11 Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96 1348-1360.   DOI
12 Fan, J. and Lv, J. (2008). Sure independence screening for ultrahigh dimensional feature space (with discussion). Journal of the Royal Statistical Society Series B, 70, 849-911.   DOI
13 Franke, B., Plante, J.-F., Roscher, R., Lee, E.-S. A., Smyth, C., Hatefi, A., Chen, F., Gil, E., Schwing, A., Selvitella, A., Hoffman, M. M., Grosse, R., Hendricks, D., and Reid, N. (2016). Statistical inference, learning and models in big data. International Statistical Review, To appear.
14 Kulldorff, M. (1997). A spatial scan statistic. Communications in Statistics: Theory and Methods, 26, 1481-1496.   DOI
15 Kim, D., Son, S.-W., and Jeong, H. (2014). Large-scale quantitative analysis of painting arts. Scientific Reports, 4, 7370.   DOI
16 Ko, S. and Won, J.-H. (2016). Processing large-scale data with Apache Spark, The Korean Journal of Applied Statistics, To appear.
17 Clauset, A. and Woodard, R. (2013). Estimating the historical and future probabilities of large terrorist events. Annals of Applied Statistics, 7, 1838-1865.   DOI
18 Bak, J. Y. and Oh, A. (2015). Five centuries of monarchy in Korea: mining the text of the annals of the Joseon dynasty. In Proceedings of the 9th SIGHUM workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities at the 53rd Annual Meeting of the Association for Computational Linguistics, 10-14.