ORIGINAL ARTICLE
Design of highly effective multilayer feedforward neural network by using genetic algorithm
Oğuz Üstün,
Erdal Bekiroğlu,
Mithat Önder,
Oğuz Üstün
Department of Electrical and Electronics Engineering, Faculty of Engineering, Bolu Abant Izzet Baysal University, Bolu, Turkey
Search for more papers by this authorErdal Bekiroğlu
Department of Electrical and Electronics Engineering, Faculty of Engineering, Bolu Abant Izzet Baysal University, Bolu, Turkey
Search for more papers by this authorCorresponding Author
Mithat Önder
Department of Electrical and Electronics Engineering, Faculty of Engineering, Bolu Abant Izzet Baysal University, Bolu, Turkey
Correspondence
Mithat Önder, Department of Electrical and Electronics Engineering, Faculty of Engineering, Bolu Abant Izzet Baysal University, Bolu 14280, Turkey.
Email: [email protected]
Search for more papers by this authorOğuz Üstün,
Erdal Bekiroğlu,
Mithat Önder,
Oğuz Üstün
Department of Electrical and Electronics Engineering, Faculty of Engineering, Bolu Abant Izzet Baysal University, Bolu, Turkey
Search for more papers by this authorErdal Bekiroğlu
Department of Electrical and Electronics Engineering, Faculty of Engineering, Bolu Abant Izzet Baysal University, Bolu, Turkey
Search for more papers by this authorCorresponding Author
Mithat Önder
Department of Electrical and Electronics Engineering, Faculty of Engineering, Bolu Abant Izzet Baysal University, Bolu, Turkey
Correspondence
Mithat Önder, Department of Electrical and Electronics Engineering, Faculty of Engineering, Bolu Abant Izzet Baysal University, Bolu 14280, Turkey.
Email: [email protected]
Search for more papers by this authorAbstract
This paper presents a highly effective and precise neural network method for choosing the activation functions (AFs) and tuning the learning parameters (LPs) of a multilayer feedforward neural network by using a genetic algorithm (GA). The performance of the neural network mainly depends on the learning algorithms and the network structure. The backpropagation learning algorithm is used for tuning the network connection weights, and the LPs are obtained by the GA to provide both fast and reliable learning. Also, the AFs of each neuron in the network are automatically chosen by a GA. The present study consists of 10 different functions to accomplish a better convergence of the desired input–output mapping. Test studies are performed to solve a set of two-dimensional regression problems for the proposed genetic-based neural network (GNN) and conventional neural network having sigmoid AFs and constant learning parameters. The proposed GNN has also been tested by applying it to three real problems in the fields of environment, medicine, and economics. Obtained results prove that the proposed GNN is more effective and reliable when compared with the classical neural network structure.
CONFLICT OF INTEREST
We declare that there is no conflict of interest in anywhere.
REFERENCES
- Angeline, P. J., Saunders, G. M., & Pollack, J. B. (1994). An evolutionary algorithm that constructs recurrent neural networks. IEEE Transactions on Neural Networks, 5(1), 54–65. https://doi.org/10.1109/72.265960
- Blanco, A., Delgado, M., & Pegalajar, M. C. (2001). A real-coded genetic algorithm for training recurrent neural networks. Neural Networks, 14(1), 93–105. https://doi.org/10.1016/S0893-6080(00)00081-2
- Center for Machine Learning and Intelligent Systems [DataSets] (2019). Retrieved from http://archive.ics.uci.edu/ml/datasets.php.
- Daqi, G., & Genxing, Y. (2003). Influences of variable scales and activation functions on the performances of multilayer multilayer feedforward neural networks. Pattern Recognition, 36(4), 869–878. https://doi.org/10.1016/S0031-3203(02)00120-6
- Detrano, R., Janosi, A., Steinbrunn, W., Pfisterer, M., Schmid, J., Sandhu, S., … Froelicher, V. (1989). International application of a new probability algorithm for the diagnosis of coronary artery disease. American Journal of Cardiology, 64, 304–310. https://doi.org/10.1016/0002-9149(89)90524-9
- Ertuğrul, Ö. F. (2018). A novel type of activation function in artificial neural networks: Trained activation function. Neural Networks, 99(1), 148–157. https://doi.org/10.1016/j.neunet.2018.01.007
- Ferreira, R. P., Martiniano, A., Ferreira, A., A., Ferreira, A., & Sassi, R.J. (2016). Study on daily demand forecasting orders using artificial neural network. IEEE Latin America Transactions, 14(3), 1519–1525. DOI: https://doi.org/10.1109/TLA.2016.7459644
- Goldberg, D. E. (1989). Genetic algorithms in search, optimization and machine learning (1st ed.). Boston, USA: Addison-Wesley. ISBN. isbn:0201157675.
- Guarnieri, S., Piazza, F., & Uncini, A. (1999). Multilayer multilayer feedforward networks with adaptive spline activation function. IEEE Transactions on Neural Networks, 10(3), 672–683. https://doi.org/10.1109/72.761726
- Hagan, M.T, Demuth, H.B., & Beale, M. (1996). Neural network design ( 2nd ed.).Texas, USA: Amazon. ISBN-10: 0–9717321–1-6
- Hardalaç, F., & Güler, G. (2008). Examination of static and 50 Hz electric fieldeffects on tissues by using a hybrid genetic algorithm and neural network. Expert Systems, 25(4), 349–366. https://doi.org/10.1111/j.1468-0394.2008.00447.x
- Haykin, S. (2008). Neural networks and learning machines ( 3rd ed. Ontario, Canada: Pearson. ISBN-13: 978–0–13-147139-9
- Hong, X. L., & Xu, L. D. (2000). A neural network representation of linear programming. European Journal of Operational Research, 124(2), 224–234. https://doi.org/10.1016/S0377-2217(99)00376-8
- Hwang, J. N. S., Lay, R., Maechler, M., Martin, R. D., & Schimert, J. (1994). Regression modeling in back-propagation and projection pursuit learning. IEEE Transactions on Neural Networks, 5(3), 342–353. https://doi.org/10.1109/72.286906
- Jackowski, K., & Wozniak, M. (2010). Method of classifier selection using the genetic approach. Expert Systems, 27(2), 114–128. https://doi.org/10.1111/j.1468-0394.2010.00513.x
- Kamiyama, N., Iijima, N., Taguchi, A., Mitsui, H., Yoshida, Y., & Sone, M. (1992). Tuning of learning rate and momentum on back-propagation. Singapore ICCS/SlTA, 92, 528–532. https://doi.org/10.1109/ICCS.1992.254895
- Kandil, N., Khorasani, K., Patel, R. V., & Sood, V. K. (1993). Optimum learning rate for backpropagation neural networks. Canadian Conference on Electrical and Computer Engineering, 1, 465–468. https://doi.org/10.1109/CCECE.1993.332193
10.1109/CCECE.1993.332193 Google Scholar
- Kathirvalavakumar, T., & Subavathi, S. J. (2009). Neighborhood based modified backpropagation algorithm using adaptive LPsfor training multilayer feedforward neural networks. Neurocomputing, 72, 16–18. https://doi.org/10.1016/j.neucom.2009.04.010
- Kwok, T. Y., & Yeung, D. Y. (1997). Objective functions for training new hidden units in constructive neural networks. IEEE Transactions on Neural Networks, 8(5), 1131–1148. https://doi.org/10.1109/72.623214
- Leung, F. H., Lam, F. H., Ling, K. S. H., & Tam, P. K. S. (2003). Tuning of the structure and parameters of a neural network using an improved genetic algorithm. IEEE Transactions on Neural Networks, 14(1), 79–88. https://doi.org/10.1109/TNN.2002.804317
- Li, F., Xu, L. D., Jin, C., & Wang, H. (2011). Intelligent bionic genetic algorithm (IB-GA) and its convergence. Expert Systems with Applications, 38(7), 8804–8811. https://doi.org/10.1016/j.eswa.2011.01.091
- Li, F., Xu, L. D., Jin, C., & Wang, H. (2012). Random assignment method based on genetic algorithms and its application in resource allocation. Expert Systems with Applications, 39(15), 12213–12219. https://doi.org/10.1016/j.eswa.2012.04.055
- Ma, L., & Khorasani, K. (2005). Constructive multilayer feedforward neural networks using hermite polynomial activation functions. IEEE Transactions on Neural Networks, 16(4), 821–833. https://doi.org/10.1109/TNN.2005.851786
- Marwala, T. (2007). Bayesian training of neural networks using genetic programming. Pattern Recognition Letters, 28(12), 1452–1458. https://doi.org/10.1016/j.patrec.2007.03.004
- Mayer, H. A., & Schwaiger, R. (2002). Differentiation of neuron types bbackpropagation neural networksal neural networks. Proceedings of the international joint conference on neural networks, 72, 1773–1778. https://doi.org/10.1109/IJCNN.2002.1007787
- Mishra, A., Chandra, P., Ghose, U. Ç., & Sodhi, S. S. (2017). Bi-modal derivative adaptive activation function sigmoidalmultilayer feedforward artificial neural networks. Applied Soft Computing, 61, 983–994. https://doi.org/10.1016/j.asoc.2017.09.002
- Nagori, V. (2016). Fine tuning the parameters of back propagation algorithm for optimum learning performance. 2nd International Conference on Contemporary Computing and Informatics (IC3I), 7–12. DOI: https://doi.org/10.1109/IC3I.2016.7917926
- Namig, J. G., & Vugar, E. I. (2018). Approximation capability of two hidden layer multilayer feedforward neural networks with fixed weights. Neurocomputing, 316, 262–269. https://doi.org/10.1016/j.neucom.2018.07.075
- Pazera, M., Buciakowski, M., Witczak, M., & Mrugalski, M. (2018). A quadratic boundedness approach to a neural network-based simultaneous estimation of actuator and sensor faults. Neural Compututing and Applications., 32, 379–389. https://doi.org/10.1007/s00521-018-3706-8
- Pedrajas, N. G., Boyer, D. O., & Martinez, C. H. (2006). An alternative approach for neural network evolution with a genetic algorithm: Crossover by combinatorial optimization. Neural Networks, 19(4), 514–528. https://doi.org/10.1016/j.neunet.2005.08.014
- Qian, S., Zhou, S., Chang, W., & Wei, F. (2017). An improved aircraft landing distance prediction model based on particle swarm optimization-extreme learning machine method. 2017 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), 2326–2330. DOI: https://doi.org/10.1109/IEEM.2017.8290307
- Sexton, R. S., & Gupta, J. N. D. (2000). Comparative evaluation of genetic algorithm and backpropagation for training neural networks. Information Sciences, 129(1–4), 45–59. https://doi.org/10.1016/S0020-0255(00)00068-2
- Sheell, S., Varshney, T., & Varshney, R. (2007). Accelerated learning in MLP using adaptive learning rate with momentum coefficients. Second International Conference on Industrial and Information Systems, 307–310. DOI: https://doi.org/10.1109/ICIINFS.2007.4579193
- Ustun, O. (2009). Determining of activation functions in a multilayer feedforward neural network by using genetic algorithm. Journal of Engineering Sciences, 15(3), 225–234.
- Ustun, O., & Yildiz, İ. (2009). Determination of the learning parameters in backpropagation learning algorithm by genetic algorithm. SDU International Technologic Science, 1(2), 61–73.
- Wang, C., Qin, S. Y., & Wan, B. W. (1991). A novel neural network structure with fast convergence based on optimizing combination of different activation function. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 13(3), 1399–1400. https://doi.org/10.1109/IEMBS.1991.684510
- Wang, P., Zhang, J., Xu, L. D., Wang, H., Feng, S., & Zhu, H. (2011). How to measure adaptation complexity in evolvable systems—A new synthetic approach of constructing fitness functions. Expert Systems with Applications, 38(8), 10414–10419. https://doi.org/10.1016/j.eswa.2011.02.099
- Winiczenko, R., Gornicki, K., Kaleta, A., & Mankowska, M. J. (2018). Optimisation of ANN topology for predicting the rehydrated apple cubes colour change using RSM and GA. Neural Compututing and Applications, 30, 1795–1809. https://doi.org/10.1007/s00521-016-2801-y
- Wong, K. W., Leung, C. S., & Chang, S. J. (2002). Use of periodic and monotonic activation functions in multilayer multilayer feedforward neural networks trained by extended Kalman filter algorithm. IEE Proceeding- Vision. Image Signal Processsing, 149(4), 217–224. https://doi.org/10.1049/ip-vis:20020515
10.1049/ip-vis:20020515 Google Scholar
- Xu, S., & Zhang, M. (2001). A novel adaptive activation function. Proceedings of the International Joint Conference on Neural Networks, 4, 2779–2782. https://doi.org/10.1109/IJCNN.2001.938813
- Yao, X. (1999). Evolving artificial neural networks. Proceedings of the IEEE, 87(9), 1423–1447. https://doi.org/10.1109/5.784219
- Yu, C. C., & Liu, B. D. (2002). A back propagation algorithm with adaptive learning rate and momentum coefficient. Proceedings of the 2002 International Joint Conference on Neural Networks, 2, 1218–1223. https://doi.org/10.1109/IJCNN.2002.1007668
- Yu, X. H., & Chen, G. A. (1997). Efficient backpropagation learning using optimal learning rate and momentum. Neural Networks, 10(3), 517–527. https://doi.org/10.1016/S0893-6080(96)00102-5
- Zaghwl, A., & Dong, W. M. (1994). An automated approach for selecting the learning rate and momentum in back-propagation networks. IEEE International Conference on Neural Networks, 1, 464–469. https://doi.org/10.1109/ICNN.1994.374207
- Zhang, S., Guo, B., Dong, A., He, J., Xu, Z., & Chen, S. X. (2017). Cautionary tales on air-quality improvement in Beijing. Proceedings of the Royal Society A, 473, 1–14. https://doi.org/10.1098/rspa.2017.0457
- Zhou, S. M., & Xu, L. D. (2001). A new type of recurrent fuzzy neural network for modelling dynamic systems. Knowledge-Based Systems, 14, 243–251. https://doi.org/10.1016/S0950-7051(01)00102-2
