Browse > Article
http://dx.doi.org/10.3741/JKWRA.2021.54.S-1.1071

A study on the development of quality control algorithm for internet of things (IoT) urban weather observed data based on machine learning  

Lee, Seung Woon (International Center for Urban Water Hydroinformatics Research & Innovation)
Jung, Seung Kwon (International Center for Urban Water Hydroinformatics Research & Innovation)
Publication Information
Journal of Korea Water Resources Association / v.54, no.spc1, 2021 , pp. 1071-1081 More about this Journal
Abstract
In addition to the current quality control procedures for the weather observation performed by the Korea Meteorological Administration (KMA), this study proposes quality inspection standards for Internet of Things (IoT) urban weather observed data based on machine learning that can be used in smart cities of the future. To this end, in order to confirm whether the standards currently set based on ASOS (Automated Synoptic Observing System) and AWS (Automatic Weather System) are suitable for urban weather, usability was verified based on SKT AWS data installed in Seoul, and a machine learning-based quality control algorithm was finally proposed in consideration of the IoT's own data's features. As for the quality control algorithm, missing value test, value pattern test, sufficient data test, statistical range abnormality test, time value abnormality test, spatial value abnormality test were performed first. After that, physical limit test, stage test, climate range test, and internal consistency test, which are QC for suggested by the KMA, were performed. To verify the proposed algorithm, it was applied to the actual IoT urban weather observed data to the weather station located in Songdo, Incheon. Through this, it is possible to identify defects that IoT devices can have that could not be identified by the existing KMA's QC and a quality control algorithm for IoT weather observation devices to be installed in smart cities of future is proposed.
Keywords
Internet of things; Weather observation; Machine learning; Quality control; Smart city;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Kim, H.S. (2020). "A study on the data quality management evaluation model." Journal of the Korea Convergence Society, Vol. 11, No. 17, pp. 217-222. (in Korean)
2 Kim, S.G., Kim, S.H., Lim, C.H., Na, S.K., Park, S.S., Kim, J.M., and Lee, Y.G. (2021). "Analysis of future demend and utilization of the urban meteorological data for the smart city." Journal of the Korea Meteorological Society, KOMES, Vol. 31, No. 2, pp. 241-249. (in Korean)
3 Korea Meteorological Administration (KMA) (2019b). Standards and procedures for quality ratings of weather observation data. pp. 4-5. (in Korean)
4 Oh, S.W., Im, N.H., Lee, S.H., and Kim, M.S. (2020). "Long-term price prediction and trend analysis of garlic using prophet model." Journal of the Korean Data Analysis Society, Vol. 22, No. 6, pp. 2325-2336. (in Korean)   DOI
5 Taylor, S.J, and Letham, B. (2017). "Forecasting at scale." PeerJ Preprints Vol. 5, e3190v2. doi: 10.7287/peerj.preprints.3190v2   DOI
6 Kim, S.Y., Lee, H.C., and Ryoo, S.B. (2016c). "Development and assessment of real-time quality control algorithm for PM10 data observed by continuous ambient particulate monitor." Journal of the Korea Meteorological Society, Atmosphere, Vol. 26, No. 4, pp. 541-551. (in Korean)
7 Kim, H.J., Lee, H.S., Choi, B.J., and Kim, Y.H. (2019). "Machine learning-based quality control and error correction using homogeneous temporal data collected by IoT sensors." Journal of the Korea Convergence Society, Vol. 10, No. 4, pp. 17-23. (in Korean)   DOI
8 Kim, J.S., Lee, D.J., Kim, J.G., Shin, C.S., Park, J.W., and Cho, Y.Y. (2016a). "Real-time weather observation system based on IoT." Annual Conference of KIPS, KIPS, pp. 926-927. (in Korean)
9 Kim, M., Lee, N.Y., and Park, J.H. (2016b). "A quality evaluation model for IoT services." Korean Information Processing Society Review, KIPS Tr. Comp. and Comm. Sys., Vol. 5, No. 9, pp. 269-274. (in Korean)
10 Korea Meteorological Administration (KMA) (2019a). Enforcement rules of the weather observation standardization act. (in Korean)
11 Miller, J.S. (1996). U.S. Patent No. 5,506,984. U.S. Patent and Trademark Office, Washington, DC, U.S.
12 Park, C.Y., and Choi, Y.E. (2012). "Validation of quality control algorithms for temperature data of the Republic of Korea." Journal of the Korea Meteorological Society, KOMES, Vol. 22, No. 3, pp. 299-307. (in Korean)
13 GTONE (2019). IoT stream data quality measurement indicators and profiling method for internet of things and system therefore. KR Patent, 1020190102059, filed Aug 21, 2019, Issued Dec 24, 2019.
14 Lee, Y.S., Kim, H.K., Hyun, M.J., Lee, H.A., Lee, Y.H., and Lee, J.W. (2017). "Derivation of standard values for quality control of temperature and precipitation observed data." Proceedings of the Autumn Meeting of KMS, KOMES, pp. 222-223. (in Korean)
15 Seo, J.H., Hwang, S.Y., and Kim, Y.H. (2002). "An implementation of radiosonde for atmospheric pressure information acquisition." Proceedings of the International Conference of Manufacturing Technology Engineers, KSMTE, pp. 618-620. (in Korean)
16 Ahn, J.H. (2009). "Development of embedded weather station using ethernet." Power Electronics Annual Conference, KIPE, pp. 13-15. (in Korean)
17 Chae, J.H., Park, M.S., and Choi, Y.J. (2014). "The WISE quality control system for integrated meteorological sensor data." Journal of the Korea Meteorological Society, KOMES, Vol. 24, No. 3, pp. 445-456. (in Korean)
18 Gartner (2018). How to create a business case for data quality improvement, accessed 14 October 2021, .
19 Jang, B.J., and Jeong, I.T. (2020). "Development of IoT sensing technology for urban weather and environment observation using the micro mobility." Proceedings of Korea Society of Civil Engineering Conference, KSCE, pp. 504-505. (in Korean)