Browse > Article
http://dx.doi.org/10.3837/tiis.2015.11.002

Traffic Offloading in Two-Tier Multi-Mode Small Cell Networks over Unlicensed Bands: A Hierarchical Learning Framework  

Sun, Youming (National Digital Switching System Engineering & Technological Research Center)
Shao, Hongxiang (College of Communications Engineering, PLA University of Science and Technology)
Liu, Xin (Comprehensive Trainning Base of Guangzhou Military Area)
Zhang, Jian (National Digital Switching System Engineering & Technological Research Center)
Qiu, Junfei (College of Communications Engineering, PLA University of Science and Technology)
Xu, Yuhua (College of Communications Engineering, PLA University of Science and Technology)
Publication Information
KSII Transactions on Internet and Information Systems (TIIS) / v.9, no.11, 2015 , pp. 4291-4310 More about this Journal
Abstract
This paper investigates the traffic offloading over unlicensed bands for two-tier multi-mode small cell networks. We formulate this problem as a Stackelberg game and apply a hierarchical learning framework to jointly maximize the utilities of both macro base station (MBS) and small base stations (SBSs). During the learning process, the MBS behaves as a leader and the SBSs are followers. A pricing mechanism is adopt by MBS and the price information is broadcasted to all SBSs by MBS firstly, then each SBS competes with other SBSs and takes its best response strategies to appropriately allocate the traffic load in licensed and unlicensed band in the sequel, taking the traffic flow payment charged by MBS into consideration. Then, we present a hierarchical Q-learning algorithm (HQL) to discover the Stackelberg equilibrium. Additionally, if some extra information can be obtained via feedback, we propose an improved hierarchical Q-learning algorithm (IHQL) to speed up the SBSs' learning process. Last but not the least, the convergence performance of the proposed two algorithms is analyzed. Numerical experiments are presented to validate the proposed schemes and show the effectiveness.
Keywords
Small cells; Stackelberg game; Q-learning; Traffic offloading;
Citations & Related Records
연도 인용수 순위
  • Reference
1 J. G. Andrews, H. Claussen, M. Dohler, et al. “Femtocells: past, present, and future,” IEEE J. Sel. Areas Commun., vol. 30, no. 3, pp. 497-508, Apr. 2012. Article (CrossRef Link)   DOI
2 J. G. Andrews, S. Buzzi, W. Choi, et al. “What will 5G be?,” IEEE J. Sel. Areas Commun., vol. 32, no. 6, pp. 1065-1082, Jun. 2014. Article (CrossRef Link)   DOI
3 Y. Xu, J. Wang, Q. Wu, et al. “A game theoretic perspective on self-organizing optimization for cognitive small cells,” IEEE Commun. Mag., vol. 53, no. 7, pp. 100-108, Jul. 2015. Article (CrossRef Link)   DOI
4 M. Jo, T. Maksymyuk, B. Strykhalyuk, et al. “Device-to-Device Based Heterogeneous Radio Access Network Architecture for Mobile Cloud Computing,” IEEE Wireless Commun., vol. 22, no.3 , pp. 50-58, Jun. 2015. Article (CrossRef Link)   DOI
5 Z. Du, Q. Wu, P. Yang, et al. “Exploiting user demand diversity in heterogeneous wireless networks,” IEEE Trans. Wireless Commun., vol. 14, no. 8, pp. 4142-4155, Aug. 2015. Article (CrossRef Link)   DOI
6 Qualcomm. "LTE in unlicensed spectrum: harmonious coexistence with Wi-Fi," white paper, Jun. 2014. Article (CrossRef Link)
7 Z. Du, Q. Wu, P. Yang, et al. “Dynamic user demand driven online network selection,” IEEE Trans. Veh. Tech. vol. 18, no. 3, pp. 1089-7798. Mar. 2014. Article (CrossRef Link)
8 F. Liu, E. Bala, E. Erkip, et al. “Small cell traffic balancing over licensed and unlicensed Bands,”IEEE Trans. Veh. Tech., to appear. Article (CrossRef Link)
9 M. Bennis, M. Simsek, A. Czylwik, et al. “When cellular meets WiFi in wireless small cell networks,” IEEE Commun. Mag., vol. 51, no. 6, pp. 44-50. Jun. 2013. Article (CrossRef Link)   DOI
10 Huawei, "U-LTE: unlicensed spectrum utilization of LTE," white paper, Feb. 2014. Article (CrossRef Link)
11 M. Mustonen, T. Chen, H. Saarnisaari, et al, “Cellular architecture enhancement for supporting the european licensed shared access concept,” IEEE Wireless Mag., vol. 21, no. 3, pp. 37-43. Jun. 2014. Article (CrossRef Link)   DOI
12 Federal Communications Commission, Unlicensed operation in the TV broadcast bands, 47 CFR Part 15, Federal Register, vol. 74, no. 30, Feb. 2009.Article (CrossRef Link)
13 M. Iwamura, K. Etemad, M.-H. Fong, R. Nory, and R. Love, “Carrier aggregation framework in 3GPP LTE-advanced [WiMAX/LTE update],” IEEE Commun. Mag., vol. 48, no. 8, pp. 60-67, Aug. 2010. Article (CrossRef Link)   DOI
14 A. al-Dulaimi, S. al-rubaye, Q. Ni, et al. “Pursuit of more capacity triggers LTE in unlicensed band,” IEEE vehicular Technology Mag., vol. 10, no. 1, pp. 43-51. 2015. Article (CrossRef Link)   DOI
15 3GPP-TSG-RAN-Meetin65, "Study on licensed assisted access using LTE," Tech. Rep. RP141664, Sep. 2014. Article (CrossRef Link)
16 Y. Xu, J. Wang, Q. Wu, et al. “Opportunistic spectrum access in cognitive radio networks: Global optimization using local interaction games,” IEEE Journal of Selected Topics in Signal Processing, vol. 6, no. 2, pp. 180-194, Apr. 2012 .Article (CrossRef Link)   DOI
17 Y. Xu, J. Wang, Q. Wu, et al. “Opportunistic spectrum access in unknown dynamic environment: A game-theoretic stochastic learning solution,” IEEE Transactions on Wireless Communications, vol. 11, no. 4, pp. 1380-1391, Apr. 2012. Article (CrossRef Link)   DOI
18 L. Zhou, M. Chen, Y. Qian, et al. “Fairness Resource Allocation in Blind Wireless Multimedia Communications,” IEEE Transactions on Multimedia, vol. 15, no. 4, pp. 946-956, Jun. 2013. Article (CrossRef Link)   DOI
19 L. Zhou, Z. Yang, Y. Wen, et al. “Resource Allocation with Incomplete Information for QoE-driven Multimedia Communications,” IEEE Transactions on Wireless Communications, vol. 12, no. 8, pp. 3733-3745, Aug. 2013. Article (CrossRef Link)   DOI
20 Y. Xu, A. Anpalagan, Q. Wu, et al. “Decision-theoretic distributed channel selection for opportunistic spectrum access: Strategies, challenges and solutions,” IEEE Commun. Surveys and Tutorials, vol. 15, no. 4, pp. 1689-1713, Fourth Quarter, 2013. Article (CrossRef Link)   DOI
21 X. Kang, R. Zhang, and M. Motani, “Price-based resource allocation for spectrum-sharing femtocell networks: a Stackelberg game approach,” IEEE J. Sel. Areas Commun., vol. 30, no. 3, pp. 538-549, Apr. 2012. Article (CrossRef Link)   DOI
22 S. Guruacharya, D. Niyato, D. I. Kim, and E. Hossain,“Hierarchical competition for downlink power allocation in OFDMA femtocell networks,” IEEE Trans. Wireless Commun., vol. 12, no. 4, pp. 1543-1553, Apr. 2013. Article (CrossRef Link)   DOI
23 Y. Chiang and W. Liao. "Genie: An optimal green policy for energy saving and traffic offloading in heterogeneous cellular networks," in Proc. of the IEEE International Conference on Communications (ICC), Budapest , Jun. 2013, pp. 6230-6234.Article (CrossRef Link)
24 X. Chen, J. Wu, Y. Cai et al. “Energy-efficiency oriented traffic offloading in wireless networks: a brief survey and a learning approach for heterogeneous cellular networks,” IEEE J. Sel. Areas Commun., vol. 33, no. 4, pp. 627-640, Apr. 2015.Article (CrossRef Link)   DOI
25 Q. Han, B. Yang, X. Wang, et al. “Hierarchical-game-based uplink power control in femtocell networks,” IEEE Trans. Veh. Technol., vol. 63, no. 6, pp. 2819-2835, Jul. 2014. Article (CrossRef Link)   DOI
26 S. Bu, F. R. Yu, Y. Cai and H. Yanikomeroglu. “Interference-aware energy-effcient resource allocation for OFDMA-based heterogeneous networks with incomplete channel state information.” IEEE Trans. Veh. Technol., vol. 64, no. 3, pp. 1036-1050, Jun. 2014. Article (CrossRef Link)   DOI
27 Y. Sun, H. Shao, J. Qiu, et al. "Capacity offloading in two-tier small cell networks over unlicensed band: a hierarchical learning framework," in Proc. of the IEEE International Conference on Wireless Communications and Signal Processing(WCSP 15), Nanjing, Oct. 2015, to appear. Article (CrossRef Link)
28 X. Kang, Y. Chia, S. Sun et al. “Mobile data offloading through a third-party WiFi access point: An operator’s perspective,” IEEE Trans. Wireless Commun., vol. 13, no. 10, pp. 5340–5351, Oct. 2014.Article (CrossRef Link)   DOI
29 L. Gao, G. Iosifidis, J. Huang, and L. Tassiulas, "Economics of mobile data offloading," in Proc. of IEEE INFOCOM SDP Workshop, Turin, Italy, Apr. 2013, pp. 3303-3308. Article (CrossRef Link)
30 J.B. Rosen. “Existence and uniqueness of equilibrium points for concave N-person games,” Ecnometrica, vol. 33, no. 3, pp. 520-534, Jul. 1965. Article (CrossRef Link)   DOI
31 D. Fudenburg and J. Tirole. “Game theory,” The MIT press, 1991. Article (CrossRef Link)
32 M. Bennis, S. M. Perlaza, P. Blasco, et al. “Self-organization in small cell networks: a reinforcement learning approach,” IEEE Trans. Wireless Commun., vol. 12, no. 7, pp. 3202-3212, Jul. 2013.Article (CrossRef Link)   DOI
33 F. Zhang, W. Zhang and Q. Ling. “Non-cooperative game for capacity offload.” IEEE Trans. Wireless Commun., vol. 11, no. 4, pp. 1565-1575, Apr. 2012. Article (CrossRef Link)   DOI
34 R. Mahindra, H. Viswanathan, K. Sundaresan, et al. "A practical traffic management system for integrated LTE-WiFi networks," in Proc. of the 20th annual international conference on Mobile computing and networking(Mobicom 14), Hawaii,USA , Sep. 2014, pp. 189-200. Article (CrossRef Link)
35 C. J. C. H. Watkins and P. Dayan. “Q-learning,” Mach. Learn., vol. 8, pp. 279-292, 1992. Article (CrossRef Link)
36 Z. Du, Q. Wu and P. Yang. “Dynamic user demand driven online network selection,” IEEE Commun. Letters, vol. 18, no. 3, pp. 419-422, Mar. 2014.Article (CrossRef Link)   DOI
37 R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 1998.Article (CrossRef Link)
38 X. Chen, H. Zhang, T. Chen, et al. "Improving energy efficiency in femtocell networks: a hierarchical reinforcement learning framework," in Proc. of the IEEE International Conference on Communications (ICC), Budapest, Jun. 2013, pp. 2241-2245.Article (CrossRef Link)
39 P. S. Sastry, V. V. Phansalkar, and M. Thathachar, “Decentralized learning of nash equilibria in multi-person stochastic games with incomplete information,” IEEE Trans. on Systems, Man and Cybernetics., vol. 24, no. 5, pp. 769-777, May. 1994. Article (CrossRef Link)   DOI
40 A. Kianercy and A. Galstyan. “Dynamics of Boltzmann Q-learning in two-player two-action games,” Physical Review E, vol. 85, no. 4, pp.1-10, Apr. 2012.Article (CrossRef Link)   DOI