Browse > Article
http://dx.doi.org/10.3837/tiis.2021.11.020

QLGR: A Q-learning-based Geographic FANET Routing Algorithm Based on Multi-agent Reinforcement Learning  

Qiu, Xiulin (School of Computer Science and Engineering, Nanjing University of Science and Technology)
Xie, Yongsheng (School of Computer Science and Engineering, Nanjing University of Science and Technology)
Wang, Yinyin (School of Computer Science and Engineering, Nanjing University of Science and Technology)
Ye, Lei (School of Computer Science and Engineering, Nanjing University of Science and Technology)
Yang, Yuwang (School of Computer Science and Engineering, Nanjing University of Science and Technology)
Publication Information
KSII Transactions on Internet and Information Systems (TIIS) / v.15, no.11, 2021 , pp. 4244-4274 More about this Journal
Abstract
The utilization of UAVs in various fields has led to the development of flying ad hoc network (FANET) technology. In a network environment with highly dynamic topology and frequent link changes, the traditional routing technology of FANET cannot satisfy the new communication demands. Traditional routing algorithm, based on geographic location, can "fall" into a routing hole. In view of this problem, we propose a geolocation routing protocol based on multi-agent reinforcement learning, which decreases the packet loss rate and routing cost of the routing protocol. The protocol views each node as an intelligent agent and evaluates the value of its neighbor nodes through the local information. In the value function, nodes consider information such as link quality, residual energy and queue length, which reduces the possibility of a routing hole. The protocol uses global rewards to enable individual nodes to collaborate in transmitting data. The performance of the protocol is experimentally analyzed for UAVs under extreme conditions such as topology changes and energy constraints. Simulation results show that our proposed QLGR-S protocol has advantages in performance parameters such as throughput, end-to-end delay, and energy consumption compared with the traditional GPSR protocol. QLGR-S provides more reliable connectivity for UAV networking technology, safeguards the communication requirements between UAVs, and further promotes the development of UAV technology.
Keywords
FANET; GPSR; dynamic environment; multi-agent reinforcement learning; local information;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Otto, A., Agatz, N., Campbell, J., Golden, B., Pesch, E., "Optimization approaches for civil applications of unmanned aerial vehicles (UAVs) or aerial drones: A survey," Special Issue on Drone Delivery Systems, 72(4), 411-458, 2018.
2 Hayat, S., Yanmaz, E., Muzaffar, R., "Survey on unmanned aerial vehicle networks for civil applications, A communications viewpoint," IEEE Communications Surveys & Tutorials, 18(4), 2624-2661, 2016.   DOI
3 Mowla, N.I., Tran, N.H., Doh, I., Chae, K., "AFRL: Adaptive federated reinforcement learning for intelligent jamming defense in FANET," Journal of Communications and Networks, 22(3), 244- 258, 2020.   DOI
4 Gankhuyag, G., Shrestha, A.P., Yoo, S.J., "Robust and Reliable Predictive Routing Strategy for Flying Ad-Hoc Networks," IEEE Access, vol. 5, pp. 643-654, 2017.   DOI
5 Arafat, M.Y., Moh, S., "Localization and Clustering Based on Swarm Intelligence in UAV Networks for Emergency Communications," IEEE Internet of Things Journal, 6(5), 8958-8976, 2019.   DOI
6 Wang, S., Huang, C., Wang, D., "Delay-aware relay selection with heterogeneous communication range in VANETs," Wireless Networks, 26(2), 995-1004, 2020.   DOI
7 Gunduz, D., De Kerret, P., Sidiropoulos, N.D., Gesbert, D., Murthy, C., Mihaela, V.D.S., "Machine Learning in the Air," IEEE Journal on Selected Areas in Communications, 37(10), 2184-2199, 2019.   DOI
8 Valadarsky, A., Schapira, M., Shahaf, D., Tamar, A., "Learning to Route," in Proc. of the 16th ACM Workshop on Hot Topics in Networks, pp. 185-191, 2017.
9 Jung, W., Yim, J., Ko, Y., "QGeo: Q-Learning-Based Geographic Ad Hoc Routing Protocol for Unmanned Robotic Networks," IEEE Communications Letters, 21(10), 2258-2261, 2017.   DOI
10 Defrawy, K.E., Tsudik, G., "ALARM: Anonymous Location-Aided Routing in Suspicious MANETs," IEEE Transactions on Mobile Computing, 10(9), 1345-1358, 2011.   DOI
11 Narayanan, P.S., Joice, C.S., "Vehicle-to-Vehicle (V2V) Communication using Routing Protocols: A Review," in Proc. of 2019 International Conference on Smart Structures and Systems (ICSSS), pp. 1-10, 2019.
12 Karp, B., Kung, H.T., "GPSR: greedy perimeter stateless routing for wireless networks," in Proc. of the 6th annual international conference on Mobile computing and networking, Boston, Massachusetts, USA, pp. 243-254, 2000.
13 Boyan, J.A., Littman, M.L., "Packet Routing in Dynamically Changing Networks: A Reinforcement Learning Approach," J.A.i.n.i.p.s., 6, 671-678, 1993.
14 Pourpeighambar, B., Dehghan, M., Sabaei, M., "Multi-agent learning based routing for delay minimization in cognitive radio networks," Journal of Network and Computer Applications, 84, 82-92, 2017.   DOI
15 Elwhishi, A., Ho, P.H., Naik, K., Shihada, B., "ARBR: Adaptive reinforcement-based routing for DTN," in Proc. of 2010 IEEE 6th International Conference on Wireless and Mobile Computing, Networking and Communications, 2010.
16 Zeng, S., Xu, X., Chen, Y., "Multi-Agent Reinforcement Learning for Adaptive Routing: A Hybrid Method using Eligibility Traces," in Proc. of IEEE 16th International Conference on Control & Automation (ICCA), pp. 1332-1339, 2020.
17 Kaviani, S., Bo, R., Ahmed, E., Larson, K.A., Kim, J.H., "Robust and Scalable Routing with MultiAgent Deep Reinforcement Learning for MANETs," arXiv preprint arXiv:2101.03273, 2021.
18 Li, R., Li, F., Li, X., Wang, Y., "QGrid: Q-learning based routing protocol for vehicular ad hoc networks," in Proc. of 2014 IEEE 33rd International Performance Computing and Communications Conference (IPCCC), pp. 1-8, 2014.
19 Kasana, R., Kumar, S., "A geographic routing algorithm based on Cat Swarm Optimization for vehicular ad-hoc networks," in Proc. of 2017 4th International Conference on Signal Processing and Integrated Networks (SPIN), pp. 86-90, 2017.
20 Mammeri, Z., "Reinforcement Learning Based Routing in Networks: Review and Classification of Approaches," IEEE Access, 7, 55916-55950, 2019.   DOI
21 Al-Rawi, H.A.A., Ng, M.A., Yau, K.-L.A., "Application of reinforcement learning to routing in distributed wireless networks: a review," Artificial Intelligence Review, 43(3), 381-416, 2015.   DOI
22 Gawlowicz, P., Zubow, A., "ns3-gym: Extending openai gym for networking research," J.a.p.a., 2018.
23 Singh, N., Elamvazuthi, I., Nallagownden, P., Ramasamy, G., Jangra, A.J.S., "Routing Based Multi-Agent System for Network Reliability in the Smart Microgrid," Sensors, 20(10), 2020.
24 Liang, X., Balasingham, I., Byun, S.-S., "A multi-agent reinforcement learning based routing protocol for wireless sensor networks," in Proc. of 2008 IEEE International Symposium on Wireless Communication Systems, pp. 552-557, 2008.
25 Hong, J., Zhang, D., Niu, X., " Impact Analysis of Node Motion on the performance of FANET routing protocols," in Proc. of 14th International Conference on Wireless Communications, Networking and Mobile Computing (WiCOM 2018), pp. 147-162, 2018.
26 Huang, H., Yin, H., Min, G., Zhang, J., Wu, Y., Zhang, X., "Energy-Aware Dual-Path Geographic Routing to Bypass Routing Holes in Wireless Sensor Networks," IEEE Transactions on Mobile Computing, 17(6), 1339-1352, 2018.   DOI
27 Mukhutdinov, D., Filchenkov, A., Shalyto, A., Vyatkin, V., "Multi-agent deep learning for simultaneous optimization for time and energy in distributed routing system," Future Generation Computer Systems, 94, 587-600, 2019.   DOI
28 Mao, H., Gong, Z., Zhang, Z., Xiao, Z., Ni, Y., "Learning multi-agent communication under limited-bandwidth restriction for internet packet routing," J.a.p.a., 2019.
29 Luong, N.C., Hoang, D.T., Gong, S., Niyato, D., Wang, P., Liang, Y., Kim, D.I., "Applications of Deep Reinforcement Learning in Communications and Networking: A Survey," IEEE Communications Surveys & Tutorials, 21(4), 3133-3174, 2019.   DOI
30 Afzal, K., Tariq, R., Aadil, F., Iqbal, Z., Ali, N., Sajid, M., "An Optimized and Efficient Routing Protocol Application for IoV," Mathematical Problems in Engineering, 9977252, 2021.
31 Li, X., Hu, X., Zhang, R., Yang, L., "Routing Protocol Design for Underwater Optical Wireless Sensor Networks: A Multiagent Reinforcement Learning Approach," IEEE Internet of Things Journal, 7(10), 9805-9818, 2020.   DOI
32 Chmaj, G., Selvaraj, H., "Distributed Processing Applications for UAV/drones: A Survey," Progress in Systems Engineering, pp 449-454, 2015.
33 Lakew, D.S., Sa'ad, U., Dao, N., Na, W., Cho, S., "Routing in Flying Ad Hoc Networks: A Comprehensive Survey," IEEE Communications Surveys & Tutorials, 22(2), 1071-1120, 2020.   DOI
34 You, X., Li, X., Xu, Y., Feng, H., Zhao, J., Yan, H., "Toward Packet Routing with Fully-distributed Multi-agent Deep Reinforcement Learning," IEEE Transactions on Systems, Man, and Cybernetics: Systems, pp. 1-14, 2020.
35 Zheng, L.-M., Li, X.-D., Li, X.-Y., "RLAR: Adaptive routing algorithm based onreinforcement learning," J.C.E., (4), 13, 2011.
36 Li, X., Hu, X., Zhang, R., Yang, L., "Routing Protocol Design for Underwater Optical Wireless Sensor Networks: A Multi-Agent Reinforcement Learning Approach," IEEE Internet of Things Journal, 7(10), 9805-9818, 2020.   DOI
37 Cruz, E.P.F.d., "A Comprehensive Survey in Towards to Future FANETs," IEEE Latin America Transactions, 16(3), 876-884, 2018.   DOI
38 Sharma, M., Singh, M., Walia, K., Kaur, K., "A Comprehensive Study of Performance Parameters for MANET, VANET and FANET," in Proc. of 2019 IEEE 10th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), pp. 0643-0646, 2019.