Journal of Communications and Networks

The Korean Institute of Commucations and Information Sciences (한국통신학회)
권호 표지
DOI QR코드

DOI QR Code

A Probabilistic Sampling Method for Efficient Flow-based Analysis

  • Jadidi, Zahra (School of Information and Communication Technology, Griffith University, QLD) ;
  • Muthukkumarasamy, Vallipuram (School of Information and Communication Technology, Griffith University, QLD) ;
  • Sithirasenan, Elankayer (School of Information and Communication Technology, Griffith University, QLD) ;
  • Singh, Kalvinder (School of Information and Communication Technology, Griffith University, QLD)
  • Received : 2015.05.15
  • Accepted : 2016.06.09
  • Published : 2016.10.31
Download PDF
⟨ Previous Next ⟩

Abstract

Network management and anomaly detection are challenges in high-speed networks due to the high volume of packets that has to be analysed. Flow-based analysis is a scalable method which reduces the high volume of network traffic by dividing it into flows. As sampling methods are extensively used in flow generators such as NetFlow, the impact of sampling on the performance of flow-based analysis needs to be investigated. Monitoring using sampled traffic is a well-studied research area, however, the impact of sampling on flow-based anomaly detection is a poorly researched area. This paper investigates flow sampling methods and shows that these methods have negative impact on flow-based anomaly detection. Therefore, we propose an efficient probabilistic flow sampling method that can preserve flow traffic distribution. The proposed sampling method takes into account two flow features: Destination IP address and octet. The destination IP addresses are sampled based on the number of received bytes. Our method provides efficient sampled traffic which has the required traffic features for both flow-based anomaly detection and monitoring. The proposed sampling method is evaluated using a number of generated flow-based datasets. The results show improvement in preserved malicious flows.

Keywords

References

  1. J. Mai, C.-N. Chuah, A. Sridharan, T. Ye, and H. Zang, "Is sampled data sufficient for anomaly detection?," in Proc. ACM SIGCOMM, 2006, pp. 165-176.
  2. P. Winter, E. Hermann, and M. Zeilinger, "Inductive intrusion detection in flow-based network data using one-class support vector machines," in Proc. IFIP NTMS, 2011, pp. 1-5.
  3. A. Sperotto and A. Pras, "Flow-based intrusion detection," in Proc. IFIP/IEEE IM, 2011, pp. 958-963.
  4. B. Li, J. Springer, G. Bebis, and M. Hadi Gunes, "A survey of network flow applications," J. Netw. Comput. Appl., vol. 36, pp. 567-581, 2013. https://doi.org/10.1016/j.jnca.2012.12.020
  5. K. Bartos and M. Rehak, "Towards efficient flow sampling technique for anomaly detection," in Proc. TMA, 2012, pp. 93-106.
  6. J. Mai, A. Sridharan, C.-N. Chuah, H. Zang, and T. Ye, "Impact of packet sampling on portscan detection," J. Sel. Areas Commun., vol. 24, pp. 2285-2298, 2006. https://doi.org/10.1109/JSAC.2006.884027
  7. The CAIDA UCSD "DDoS Attack 2007" Dataset, [Online]. Available: http://www.caida.org/data/passive/ddos-200708nct04_dataset.xml
  8. The CAIDA UCSD Anonymized Internet Traces 2013, [Online]. Available: http://www.caida.org/data/passive/passive_2013_dataset.xml
  9. The CAIDA UCSD Anonymized Internet Traces 2012, [Online]. Available: http://www.caida.org/data/passive/passive_2012_dataset.xml
  10. A. Sperotto et al., "An overview of IP flow-based intrusion detection," IEEE Commun. Surveys Tuts., vol. 12, pp. 343-356, 2010. https://doi.org/10.1109/SURV.2010.032210.00054
  11. Z. Jadidi, V. Muthukkumarasamy, and E. Sithirasenan, "Metaheuristic algorithms based flow anomaly detector," in Proc. APCC, 2013, pp. 717-722.
  12. Z. Jadidi, V. Muthukkumarasamy, E. Sithirasenan, and M. Sheikhan, "Flow-based anomaly detection using neural network optimized with GSA algorithm," in Proc. IEEE NFSP, 2013, pp.76-81.
  13. M. Sheikhan and Z. Jadidi, "Flow-based anomaly detection in high-speed links using modified GSA-optimized neural network," Neural Comput. Appl., vol. 24, pp. 599-611, 2014. https://doi.org/10.1007/s00521-012-1263-0
  14. P. Gogoi, D. Bhattacharyya, B. Borah, and J. K. Kalita, "MLH-IDS: A multi-level hybrid intrusion detection method," The Computer Journal, vol. 57, pp. 602-623, 2014. https://doi.org/10.1093/comjnl/bxt044
  15. N. Hohn and D. Veitch, "Inverting sampled traffic," IEEE/ACM Trans. Netw., vol. 14, pp. 68-80, 2006. https://doi.org/10.1109/TNET.2005.863456
  16. N. Duffield, C. Lund, and M. Thorup, "Estimating flow distributions from sampled flow statistics," IEEE/ACM Trans. Netw., vol. 13, pp. 933-946, 2005. https://doi.org/10.1109/TNET.2005.852874
  17. B.-Y. Choi, J. Park, and Z.-L. Zhang, "Adaptive packet sampling for accurate and scalable flow measurement," in Proc. IEEE GLOBECOM, 2004, pp. 1448-1452.
  18. N. Duffield, C. Lund, and M. Thorup, "Properties and prediction of flow statistics from sampled packet streams," in Proc. ACM SIGCOMM, 2002, pp. 159-171.
  19. C. Estan and G. Varghese, "New directions in traffic measurement and accounting," in Proc. ACM SIGCOMM, vol. 32, 2002.
  20. G. Androulidakis, V. Chatzigiannakis, and S. Papavassiliou, "Network anomaly detection and classification via opportunistic sampling," IEEE Netw., vol. 23, pp. 6-12, 2009.
  21. G. Androulidakis and S. Papavassiliou, "Improving network anomaly detection via selective flow-based sampling," IET Commun., vol. 2, pp. 399-409, 2008. https://doi.org/10.1049/iet-com:20070231
  22. V. Carela-Espanol, P. Barlet-Ros, A. Cabellos-Aparicio, and J. Sole-Pareta, "Analysis of the impact of sampling on NetFlow traffic classification," Computer Netw., vol. 55, pp. 1083-1099, 2011. https://doi.org/10.1016/j.comnet.2010.11.002
  23. Z. Jadidi, V. Muthukkumarasamy, E. Sithirasenan, and K. Singh, "Performance of flow-based anomaly detection in sampled traffic," J. Netw., vol. 10, pp. 512-520, 2016.
  24. Z. Jadidi, V. Muthukkumarasamy, E. Sithirasenan, and K. Singh, "Intelligent sampling using an optimized neural network," J. Netw., vol. 11, pp. 16-27, 2016.
  25. Q. A. Tran, F. Jiang, and J. Hu, "A real-time netflow-based intrusion detection system with improved BBNN and high-frequency field programmable gate arrays," in Proc. IEEE TrustCom, 2012, pp. 201-208.
  26. [Online]. Available: http://www.mindrot.org/projects/softflowd/, as of June 2014.
  27. [Online]. Available: http://www.mindrot.org/projects/flowd/, as of June 2014.
  28. T. Qin, X. Guan, W. Li, P. Wang, and M. Zhu, "A new connection degree calculation and measurement method for large scale network monitoring," J. Netw. Comput. Appl., vol. 41, pp. 15-26, 2014. https://doi.org/10.1016/j.jnca.2013.10.008
  29. I. Paredes-Oliva, P. Barlet-Ros, and J. Sole-Pareta, "Scan detection under sampling: A new perspective," Computer, vol. 46, pp. 38-44, 2013.
  30. G. Androulidakis and S. Papavassiliou, "Intelligent flow-based sampling for effective network anomaly detection," in Proc. IEEE GLOBECOM, 2007, pp. 1948-1953.