Browse > Article
http://dx.doi.org/10.9766/KIMST.2022.25.4.372

FAST Design for Large-Scale Satellite Image Processing  

Lee, Youngrim (The Defense AI Technology Center, Agency for Defense Development)
Park, Wanyong (The Defense AI Technology Center, Agency for Defense Development)
Park, Hyunchun (The Defense AI Technology Center, Agency for Defense Development)
Shin, Daesik (The Defense AI Technology Center, Agency for Defense Development)
Publication Information
Journal of the Korea Institute of Military Science and Technology / v.25, no.4, 2022 , pp. 372-380 More about this Journal
Abstract
This study proposes a distributed parallel processing system, called the Fast Analysis System for remote sensing daTa(FAST), for large-scale satellite image processing and analysis. FAST is a system that designs jobs in vertices and sequences, and distributes and processes them simultaneously. FAST manages data based on the Hadoop Distributed File System, controls entire jobs based on Apache Spark, and performs tasks in parallel in multiple slave nodes based on a docker container design. FAST enables the high-performance processing of progressively accumulated large-volume satellite images. Because the unit task is performed based on Docker, it is possible to reuse existing source codes for designing and implementing unit tasks. Additionally, the system is robust against software/hardware faults. To prove the capability of the proposed system, we performed an experiment to generate the original satellite images as ortho-images, which is a pre-processing step for all image analyses. In the experiment, when FAST was configured with eight slave nodes, it was found that the processing of a satellite image took less than 30 sec. Through these results, we proved the suitability and practical applicability of the FAST design.
Keywords
FAST; Distributed Parallel Processing; Large-Scale Image Management; HDFS; Apache Spark; Docker;
Citations & Related Records
연도 인용수 순위
  • Reference
1 The Apache Software Foundation, "Apache Hadoop," Apache Hadoop, 2006, https://hadoop.apache.org/, accessed 22 Apr 2022.
2 Ghemawat S, Gobioff H and Leung ST, The Google File System, SIGOPS Operating Systems Review, Vol. 37, No. 5, pp. 29-43, 2003.   DOI
3 Hunt P, Konar M, Junqueira FP and Reed B, ZooKeeper: Wait-Free Coordination for Internet-Scale Systems, USENIX Annual Technical Conference, pp. 145-158, 2020.
4 Statistics Korea, "The land area(G20)," Korean Statistical Information Service, 30 Sep 2021, https://kosis.kr/statHtml/statHtml.do?orgId=101&tblId=DT_2KAA101_G20&conn_path=I2, accessed 21 July 2022.
5 Vavilapalli VK, et. al., Apache Hadoop Yarn: Yet Another Resource Negotiator, Proceedings of the 4th Annual Symposium on Cloud Computing(SOCC '13), Article 5, pp. 1-16, 2013.
6 Almeer MH, Cloud Hadoop Map Reduce for Remote Sensing Image Analysis, Journal of Emerging Trends in Computing and Information Sciences, Vol. 3, No. 4, pp. 637-644, 2012.
7 Chen Z, Chen N and Yang C, Di L, Cloud Computing Enabled Web Processing Service for Earth Observation Data Processing, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 5, No. 6, pp. 1637-1649, 2012.   DOI
8 Dean J and Ghemawat S, MapReduce: Simplified Data Processing on Large Clusters, Communications of the ACM, Vol. 51, No. 1, pp. 107-113, 2008.   DOI
9 Hee-Jeong H., Hyun Y., Jae-Moo H. and Young-Je P., "Systemic Design and Development of the Second Geostationary Ocean Color Ground Segment," KIISE Transactions on Computing Practices, Vol. 25, No. 10, pp. 477-484, 2019.   DOI
10 Huo Y, Blaber J, Damon SM, Boyd BD, Bao S, Parvathaneni P, Noguera CB, Chaganti S, Nath V, Greer JM, Lyu I, French WR, Newton AT, Rogers BP and Landman BA, Towards Portable LargeScale Image Processing with High-Performance Computing, Journal of Digital Imaging, Vol. 31, No. 3, pp. 304-314, 2018.   DOI
11 Knoth C and Nust D, Reproducibility and Practical Adoption of Geobia with Open-Source Software in Docker Containers, Remote Sensing, Vol. 9, No. 3, p. 290, 2017.
12 Kreps J, Narkhede N and Rao J, Kafka: A Distributed Messaging System for Log Processing, Proceedings of the NetDB, Vol. 11, pp. 1-7, 2011.
13 Liu X, Iftikhar N and Xie X, Survey of Real-Time Processing Systems for Big Data, Proceedings of the 18th International Database Engineering & Applications Symposium, pp. 356-361, 2014.
14 Nibedita Mohanta, "How Many Satellites are Orbiting the Earch in 2021?," Geospatial World, 28 May 2021, https://www.geospatialworld.net/blogs/howmany-satellites-are-orbiting-the-earth-in-2021/, accessed 30 Jun 2022.
15 OGC, OpenGIS Web Processing Service Specification (Version 1.0.0) OGC Standard No. 05-007r7, OGC, 2007.
16 Planet Labs PBC, "Planet," Planet, http://planet.com, accessed 22 Apr 2022
17 Rathore MM, Son H, Ahmad A, Paul A and Jeon G, Real-Time Big Data Stream Processing Using GPU with Spark Over Hadoop Ecosystem, International Journal of Parallel Programming, Vol. 46, No. 3, pp. 630-646, 2018.   DOI
18 Merkel D, Docker: Lightweight Linux Containers for Consistent Development and Deployment, Linux Journal, LLC 239, Article 2, 2014.
19 SERGEY S., TAMARA D., ANNA K. and EIDELSTEINC L., "Engaging the Public to Fight the Consequences of Terrorism and Disasters," IOS Press, 120, pp. 91-103, 2015.
20 S. N. K. B. Amit, S. Shiraishi, T. Inoshita and Y. Aoki, "Analysis of Satellite Images for Disaster Detection," 2016 IEEE International Geoscience and Remote Sensing Symposium(IGARSS), pp. 5189-5192, 2016, doi: 10.1109/IGARSS.2016.7730352.   DOI
21 Sharma T, Shokeen V and Mathur S, Distributed Processing of Satellite Images on Hadoop to Generate Normalized Difference Vegetation Index Images, Proceedings of 2017 International Conference on Computing, Communication, pp. 1-5, 2017.
22 Zaharia M, et. al., Spark: Cluster Computing with Working Sets. Proceedings of 2nd USENIX Conf. Hot Topics Cloud Comput., pp. 10-10, 2010.
23 Zhang Y, Gao Q, Gao L and Wang C, iMapReduce: A Distributed Computing Framework for Iterative Computation, Journal of Grid Computing, Vol. 10, No. 1, pp. 47-68, 2012.   DOI
24 Huang W, Meng L, Zhang D and Zhang W, In-Memory Parallel Processing of Massive Remotely Sensed Data Using an Apache Spark on Hadoop YARN Model, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 10, No. 1, pp. 3-19, 2016.   DOI
25 S. P. R and R. U, "A Survey on Agriculture Monitoring with Satellite and Its Benefits," 2022 8th International Conference on Advanced Computing and Communication Systems(ICACCS), pp. 1854-1858, 2022, doi: 10.1109/ICACCS54159.2022.9785323.   DOI