Browse > Article
http://dx.doi.org/10.14481/jkges.2015.16.5.43

Outlier Detection in Time Series Monitoring Datasets using Rule Based and Correlation Analysis Method  

Jeon, Jesung (Department of Construction Information Engineering, Induk University)
Koo, Jakap (Department of Civil, Safety & Environmental Engineering, Hankyong National University)
Park, Changmok (Department of Technology & Systems Management, Induk University)
Publication Information
Journal of the Korean GEO-environmental Society / v.16, no.5, 2015 , pp. 43-53 More about this Journal
Abstract
In this study, detection methods of outlier in various monitoring data that fit into big data category were developed and outlier detections were conducted for both artificial data and real field monitoring data. Rule-based methods applied rate of change and probability of error for monitoring data are effective to detect a large-scale short faults and constant faults having no change within a certain period. There are however, problems with misjudgement that consider the normal data with a large scale variation as outlier caused by using independent single dataset. Rule-based methods for noise faults detection have a limit to application of real monitoring data due to the problem with a choice of proper window size of data and finding of threshold for outlier judgment. A correlation analysis among different two datasets were very effective to detect localized outlier and abnormal variation for short and long-term monitoring dataset if reasonable range of training data could be selected.
Keywords
Monitoring data; Instrumentation; Outlier; Outlier detection; Rule-based method; Correlation analysis; Big data;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 Elnahrawy, E. and Nath, B. (2003), Cleaning and querying noisy sensors, Proc. of 2nd ACM International Conference on Wireless Sensor Networks and Applications, USA, pp. 78-87.
2 Jeffery, S. R., Alonso, G., Franklin, M. J., Hong, W. and Widom, J. (2006), Declarative support for sensor data cleaning, Proc. of 4th International Conference on Pervasive Computing, Ireland, pp. 83-100.
3 Kailath, T. (1975), Square-root algorithms for least-squares estimation, IEEE Trans. Automatic Control, Vol. 20, No. 4, pp. 487-497.   DOI
4 Krishnamachari, B. and Iyengar, S. (2004), Distributed bayesian algorithms for fault-tolerant event region detection in wireless sensor networks, IEEE Trans. Vol. 53, No. 3, pp. 241-250.
5 Mourad, M. and Bertrand-Krajewski, J. L. (2002), A method for automatic validation of long time series of data in urban hydrology, Water Science & Technology, Vol. 45, No. 4-5, pp. 263-270.
6 Ni, K., Ramanathan, N., Chehade, M., Balzano, L., Nair, S., Zahedi, S., Pottie, G., Hansen, M. and Srivastava., M. (2009), Sensor network data fault types, ACM Transactions on Sensor Networks, Vol. 5, No. 3, Article25, pp. 1-29.
7 Park, H. C., Hwang, H. J. and Lee, J. W. (2012), Development of new data analysis method to evaluate reliability of the sensor or measured data, Journal of the Korea Institute for Structural Maintenance and Inspection, Vol. 16, No. 6, pp. 34-44.   DOI
8 Ramanathan, N., Balzano, L., Burt, M., Estrin, D., Kohler, E., Harmon, T., Harvey, C., Jay, J., Rothenberg, S. and Srivastava, M. (2006), Rapid deployment with confidence: calibration and fault detection in environmental sensor networks. Tech. Rep. 62, CENS. pp. 1-14.
9 Sharma, A. B., Golubchik, L. and Govindan, R. (2010), Sensor faults: detection methods and prevalence in real-world datasets, ACM Transactions on Sensor Networks, Vol. 6, No. 3, Article23. pp. 1-39.
10 Szewczyk, R., Mainwaring, A., Polastre, J., Anderson, J. and Culler, D. (2004), An analysis of a large scale habitat monitoring application, Proc. of the 2nd international conference on Embedded networked sensor systems, USA, pp. 214-226.
11 Tolle, G., Polastre, J., Szewczyk, R., Culler, D., Turner, N., Tu, K., Burgess, S., Dawson, T., Buonadonna, P., Gay, D. and Hong, W. (2005), A macroscope in the redwoods, Proc. of the 2nd International Conference on Embedded Networked Sensor Systems, ACM Press, New York, pp. 51-63.
12 Williams, G. J., Baxter, R. A., He, H. X., Hawkins, S. and Gu, L. (2002), A comparative study of RNN for outlier detection in data mining, IEEE International Conference on Data-mining (ICDM'02), Maebashi City, Japan, CSIRO Technical Report CMIS-02/102, pp. 1-709.
13 Werner-Allen, G., Lorincz, K., Johnson, J., Lees, J. and Welsh, M. (2006), Fidelity and yield in a volcano monitoring sensor network, Proc. of the 7th USENIX Symposium on Operating Systems Design and Implementation, Seattle, pp. 381-396.