Reinforcement learning–based QoS/QoE-aware service function chaining in software-driven 5G slices
Xi Chen
School of Computer Science and Technology, Southwest Minzu University, Chengdu, China
Center for Cyber Security, University of Electronic Science and Technology of China (UESTC), Chengdu, China
Search for more papers by this authorZonghang Li
Center for Cyber Security, University of Electronic Science and Technology of China (UESTC), Chengdu, China
Search for more papers by this authorYupeng Zhang
Center for Cyber Security, University of Electronic Science and Technology of China (UESTC), Chengdu, China
Search for more papers by this authorRuiming Long
Center for Cyber Security, University of Electronic Science and Technology of China (UESTC), Chengdu, China
Search for more papers by this authorCorresponding Author
Hongfang Yu
Center for Cyber Security, University of Electronic Science and Technology of China (UESTC), Chengdu, China
Hongfang Yu, School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China.
Email: [email protected]
Present Address
School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China.
Search for more papers by this authorXiaojiang Du
Department of Computer and Information Sciences, Temple University, Philadelphia, Pennsylvania
Search for more papers by this authorMohsen Guizani
Department of Electrical and Computer Engineering, University of Idaho, Moscow, Idaho
Search for more papers by this authorXi Chen
School of Computer Science and Technology, Southwest Minzu University, Chengdu, China
Center for Cyber Security, University of Electronic Science and Technology of China (UESTC), Chengdu, China
Search for more papers by this authorZonghang Li
Center for Cyber Security, University of Electronic Science and Technology of China (UESTC), Chengdu, China
Search for more papers by this authorYupeng Zhang
Center for Cyber Security, University of Electronic Science and Technology of China (UESTC), Chengdu, China
Search for more papers by this authorRuiming Long
Center for Cyber Security, University of Electronic Science and Technology of China (UESTC), Chengdu, China
Search for more papers by this authorCorresponding Author
Hongfang Yu
Center for Cyber Security, University of Electronic Science and Technology of China (UESTC), Chengdu, China
Hongfang Yu, School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China.
Email: [email protected]
Present Address
School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China.
Search for more papers by this authorXiaojiang Du
Department of Computer and Information Sciences, Temple University, Philadelphia, Pennsylvania
Search for more papers by this authorMohsen Guizani
Department of Electrical and Computer Engineering, University of Idaho, Moscow, Idaho
Search for more papers by this authorAbstract
With the ever-growing diversity of devices and applications that will be connected to 5G networks, flexible and agile service orchestration with acknowledged quality of experience (QoE) that satisfies the end user's functional and quality-of-service (QoS) requirements is necessary. Software-defined networking (SDN) and network function virtualization (NFV) are considered key enabling technologies for 5G core networks. In this regard, this paper proposes a reinforcement learning–based QoS/QoE-aware service function chaining (SFC) scheme in SDN/NFV-enabled 5G slices. First, it implements a lightweight QoS information collector based on the Link Layer Discovery Protocol, which works in a piggyback fashion on the southbound interface of the SDN controller, to enable QoS-awareness. Then, a deep Q-network–based orchestration agent is designed to support SFC in the context of NFV. The agent takes into account the QoE and QoS as key aspects to formulate the reward so that it is expected to maximize QoE while respecting QoS constraints. The experiment results show that the proposed framework exhibits good performance in QoE provisioning and QoS requirements maintenance for SFC in dynamic network environments.
Supporting Information
The following supporting information is available as part of the online article:
Dataset S1. The source code and experiment results of the DQN-based QoS/QoE-aware service function chaining algorithm are available through URL: https://github.com/Lizonghang/sfc-dqn.
| Filename | Description |
|---|---|
| ETT_3477-Sup-0001-ETT-18-0051.R1-Supplementary-Material.zipapplication/x-zip-compressed, 2.4 MB |
ETT_3477-Sup-0001-ETT-18-0051.R1-Supplementary-Material.zip |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
REFERENCES
- 1Huang L, Zhu G, Du X. Cognitive femtocell networks: an opportunistic spectrum access for future indoor wireless coverage. IEEE Wirel Commun. 2013; 20(2): 44-51.
- 2Du X, Lin F. Maintaining differentiated coverage in heterogeneous sensor networks. EURASIP J Wirel Commun Netw. 2005; 5(4): 565-572.
- 3Du X, Xiao Y, Guizani M, Chen H-H. An effective key management scheme for heterogeneous sensor networks. Ad Hoc Netw. 2007; 5(1): 24-34.
- 4Al-Hammouri AT, Al-Ali Z, Al-Duwairi B. ReCAP: a distributed CAPTCHA service at the edge of the network to handle server overload. Trans Emerg Telecommun Technol. 2018; 29(4): e3187.
- 5Farris I, Taleb T, Flinck H, Iera A. Providing ultra-short latency to user-centric 5G applications at the mobile network edge. Trans Emerg Telecommun Technol. 2018; 29(4): e3169.
- 6Orsini G, Bade D, Lamersdorf W. CloudAware: empowering context-aware self-adaptation for mobile applications. Trans Emerg Telecommun Technol. 2017; 29(4): e3210.
- 7Zhang H, Chen S, Li X, Ji H, Du X. Interference management for heterogeneous networks with spectral efficiency improvement. IEEE Wirel Commun. 2015; 22(2): 101-107.
- 8Du X. QoS routing based on multi-class nodes for mobile ad hoc networks. Ad Hoc Netw. 2004; 2(3): 241-254.
10.1016/j.adhoc.2004.03.004 Google Scholar
- 9Dong P, Du X, Zheng T, Zhang H. Improving QoS on high-speed vehicle by multipath transmission based on practical experiment. Paper presented at: 2015 IEEE Vehicular Networking Conference (VNC); 2015; Kyoto, Japan.
- 10 NEC Corporation. Network Evolution Toward 2020 and Beyond. White Paper. 2015.
- 11Halpern J, Pignataro C. Service function chaining (SFC) architecture. IETF RFC. 2015; 10.
- 12Mach P, Becvar Z. Mobile edge computing: a survey on architecture and computation offloading. IEEE Commun Surv Tutor. 2017; 19(3): 1628-1656.
- 13McKeown N, Anderson T, Balakrishnan H, et al. OpenFlow: enabling innovation in campus networks. ACM SIGCOMM Comput Commun Rev. 2008; 38(2): 69-74.
- 14Khan A, Zugenmaier A, Jurca D, Kellerer W. Network virtualization: a hypervisor for the Internet? IEEE Commun Mag. 2012; 50(1): 136-143.
- 15Sutton RS, Barto AG. Reinforcement Learning: An Introduction. Cambridge, MA: The MIT Press; 2017.
- 16Jiang J, Sun S, Sekar V, Zhang H. Pytheas: enabling data-driven quality of experience optimization using group-based exploration-exploitation. In: Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI); 2017; Boston, MA.
- 17Shi R, Zhang J, Chu W, et al. MDP and machine learning-based cost-optimization of dynamic resource allocation for network function virtualization. Paper presented at: 2015 IEEE International Conference on Services Computing (SCC); 2015; New York, NY.
- 18Mohammadi M, Al-Fuqaha A, Guizani M, Oh JS. Semisupervised deep reinforcement learning in support of IoT and smart city services. IEEE Internet of Things J. 2018; 5(2): 624-635.
- 19Salahuddin MA, Al-Fuqaha A, Guizani M. Reinforcement learning for resource provisioning in the vehicular cloud. IEEE Wirel Commun. 2016; 23(4): 128-135.
- 20 Cisco. LLDP-MED and Cisco Discovery Protocol. White Paper. 2006; 1-12.
- 21Zhang Y, Beheshti N, Beliveau L, et al. StEERING: A software-defined networking for inline service chaining. Paper presented at: 2013 21st IEEE International Conference on Network Protocols (ICNP); 2013; Goettingen, Germany.
- 22Qazi ZA, Tu C-C, Chiang L, Miao R, Sekar V, Yu M. SIMPLE-fying middlebox policy enforcement using SDN. ACM SIGCOMM Comput Commun Rev. 2013; 43(4): 27-38.
- 23Fayazbakhsh SK, Sekar V, Yu M, Mogul JC. FlowTags: Enforcing network-wide policies in the presence of dynamic middlebox actions. In: Proceedings of the Second ACM SIGCOMM Workshop on Hot Topics in Software Defined Networking (HotSDN); 2013; Hong Kong, China.
- 24Fayazbakhsh SK, Chiang L, Sekar V, Yu M, Mogul JC. Enforcing network-wide policies in the presence of dynamic middlebox actions using FlowTags. In: Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI); 2013; Seattle, WA.
- 25Arumaithurai M, Chen J, Monticelli E, Fu X, Ramakrishnan KK. Exploiting ICN for flexible management of software-defined networks. In: Proceedings of the 1st ACM Conference on Information-Centric Networking (ACM-ICN); 2014; Paris, France.
- 26Kulkarni SG, Arumaithurai M, Tasiopoulos A, et al. Name enhanced SDN framework for service function chaining of elastic network functions. Paper presented at: 2016 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS); 2016; San Francisco, CA.
- 27Guo L, Pang J, Walid A. Dynamic service function chaining in SDN-enabled networks with middleboxes. Paper presented at: 2016 IEEE 24th International Conference on Network Protocols (ICNP); 2016; Singapore.
- 28Wen T, Yu H, Sun G, Liu L. Network function consolidation in service function chaining orchestration. Paper presented at: 2016 IEEE International Conference on Communications (ICC); 2016; Kuala Lumpur, Malaysia.
- 29Yang K, Zhang H, Hong P. Energy-aware service function placement for service function chaining in data centers. Paper presented at: 2016 IEEE Global Communications Conference (GLOBECOM); 2016; Washington, DC.
- 30Wen T, Yu H, Du X. Performance guarantee aware orchestration for service function chains with elastic demands. Paper presented at: 2017 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN); 2017; Berlin, Germany.
- 31Ding W, Yu H, Luo S. Enhancing the reliability of services in NFV with the cost-efficient redundancy scheme. Paper presented at: 2017 IEEE International Conference on Communications (ICC); 2017; Paris, France.
- 32Doshi P, Goodwin R, Akkiraju R, Verma K. Dynamic workflow composition using Markov decision processes. In: Proceedings. IEEE International Conference on Web Services (ICWS); 2004; San Diego, CA.
- 33Lin S-C, Akyildiz IF, Wang P, Luo M. QoS-aware adaptive routing in multi-layer hierarchical software defined networks: A reinforcement learning approach. Paper presented at: 2016 IEEE International Conference on Services Computing (SCC); 2016; San Francisco, CA.
- 34Yousaf FZ, Bredel M, Schaller S, Schneider F. NFV and SDN – key technology enablers for 5G networks. IEEE J Sel Areas Commun. 2017; 35(11): 2468-2478.
- 35Sekar V, Egi N, Ratnasamy S, Reiter MK, Shi G. Design and implementation of a consolidated middlebox architecture. In: Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI); 2012; San Jose, CA.
- 36 ITU-T . Methods for subjective determination of transmission quality - series p: telephone transmission quality; Methods for objective and subjective assessment of quality. Recommendation. 1996.
- 37Reichl P, Egger S, Schatz R, D'Alconzo A. The logarithmic nature of QoE and the role of the Weber-Fechner law in QoE assessment. Paper presented at: 2010 IEEE International Conference on Communications (ICC); 2010; Cape Town, South Africa.
- 38Fiedler M, Hossfeld T, Tran-Gia P. A generic quantitative relationship between quality of experience and quality of service. IEEE Netw. 2010; 24(2): 36-41.
- 39Watkins CJCH, Dayan P. Q-learning. Mach Learn. 1992; 8(3-4): 279-292.
- 40Mnih V, Kavukcuoglu K, Silver D, et al. Playing Atari with deep reinforcement learning. arXiv:13125602 [cs]; 2013.
- 41Auer P, Cesa-Bianchi N, Fischer P. Finite-time analysis of the multiarmed bandit problem. Mach Learn. 2002; 47(2-3): 235-256.
- 42Weber R. On the Gittins index for multiarmed bandits. Ann Appl Probab. 1992; 2(4): 1024-1033.
10.1214/aoap/1177005588 Google Scholar
