ORIGINAL ARTICLE
Automated test design using swarm and evolutionary intelligence algorithms
Muhammet Aktaş,
Zeki Yetgin,
Fatih Kılıç,
Önder Sünbül,
Corresponding Author
Muhammet Aktaş
Department of Computer Engineering, Adana Alparslan Turkes Science and Technology University, Adana, Turkey
Correspondence
Muhammet Aktaş, Department of Computer Engineering, Adana Alparslan Turkes Science and Technology University, Adana, Turkey.
Email: [email protected]
Search for more papers by this authorZeki Yetgin
Department of Computer Engineering, Mersin University, Mersin, Turkey
Search for more papers by this authorFatih Kılıç
Department of Computer Engineering, Adana Alparslan Turkes Science and Technology University, Adana, Turkey
Search for more papers by this authorÖnder Sünbül
Department of Measurement and Evaluation in Education, Mersin University, Mersin, Turkey
Search for more papers by this authorMuhammet Aktaş,
Zeki Yetgin,
Fatih Kılıç,
Önder Sünbül,
Corresponding Author
Muhammet Aktaş
Department of Computer Engineering, Adana Alparslan Turkes Science and Technology University, Adana, Turkey
Correspondence
Muhammet Aktaş, Department of Computer Engineering, Adana Alparslan Turkes Science and Technology University, Adana, Turkey.
Email: [email protected]
Search for more papers by this authorZeki Yetgin
Department of Computer Engineering, Mersin University, Mersin, Turkey
Search for more papers by this authorFatih Kılıç
Department of Computer Engineering, Adana Alparslan Turkes Science and Technology University, Adana, Turkey
Search for more papers by this authorÖnder Sünbül
Department of Measurement and Evaluation in Education, Mersin University, Mersin, Turkey
Search for more papers by this authorAbstract
The world's increasing dependence on computer-assisted education systems has raised significant challenges about student assessment methods, such as automated test design. The exam questions should test the students' potential from various aspects, such as their intellectual and cognitive levels, which can be defined as attributes of the questions to assess student knowledge. Test design is challenging when various question attributes, such as category, learning outcomes, difficulty, and so forth, are considered with the exam constraints, such as exam difficulty and duration. In this paper, four contributions are provided to overcome test design challenges for the student assessment. First, a tool is developed to generate a synthetic question pool. Second, an objective function is designed based on the considered attributes. Third, the popular swarm and evolutionary optimization methods, namely particle swarm optimization, genetic algorithm, artificial bee colony, differential search algorithm are comparatively studied with novel methodologies applied to them. Finally, as the state of the art methods, artificial bee colony, and differential search algorithm are further modified to improve the solution of the test design. To perform the proposed algorithms, a dataset of 1000 questions is built with the proposed question attributes of the test design. Algorithms are evaluated in terms of their successes in both minimizing the objective function and running time. Additionally, Friedman's test and Wilcoxon rank-sum statistical tests are applied to statistically compare the algorithms' performances. The results show that the improved artificial bee colony and the improved differential search provide better results than others in terms of optimization error and running time.
CONFLICT OF INTEREST
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Open Research
DATA AVAILABILITY STATEMENT
The data that support the findings of this study are available from the corresponding author upon reasonable request.
REFERENCES
- Abdollahzadeh, B., Gharehchopogh, F. S., & Mirjalili, S. (2021). African vultures optimization algorithm: A new nature-inspired metaheuristic algorithm for global optimization problems. Computers & Industrial Engineering, 158, 107408.
- Abdollahzadeh, B., Soleimanian Gharehchopogh, F., & Mirjalili, S. (2021). Artificial gorilla troops optimizer: A new nature-inspired metaheuristic algorithm for global optimization problems. International Journal of Intelligent Systems, 36(10), 5887–5958.
- Abduljabbar, D. A., & Omar, N. (2015). Exam questions classification based on Bloom's Taxonomy cognitive level using classifiers combination. Journal of Theoretical and Applied Information Technology, 78(3), 447.
- Abualigah, L., Yousri, D., Abd Elaziz, M., Ewees, A. A., Al-qaness, M. A., & Gandomi, A. H. (2021). Aquila optimizer: A novel meta-heuristic optimization algorithm. Computers & Industrial Engineering, 157, 107250.
- Aktaş, M. (2021). Dataset for test exam design. https://drive.google.com/file/d/1Sa5py5WAiutXD2yrvRymQ_zW1PjRILTd/view?usp=sharing
- Aktaş, M., Kılıç, F. & Yetkin, Z. (2019). Bloom's taksonomi tabanlı genetik algoritma kullanılarak otomatik sınav sorusu üretilmesi. International Congress on Humancomputer Interaction, Optimization and Robotic Applications July (pp. 5–7).
- Álvarez, P. & Baldassarri, S. (2018). Semantics and service technologies for the automatic generation of online mcq tests. 2018 IEEE Global Engineering Education Conference (EDUCON) (p. 421–426). https://doi.org/10.1109/EDUCON.2018.8363261
- Alves, C., Gierl, M., & Lai, H. (2010). Using automated item generation to promote principled test design and development. The Annual Meeting of the American Educational Research Association, Denver, CO.
- Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning teaching and assessing: A revision of Bloom's Taxonomy of educational objectives. Longman.
- Bloom, B. S., Englehart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). The Taxonomy of educational objectives, handbook I: The Cognitive domain (Vol. 1, pp. 20–24). New York: McKay.
- Bui, K.-H. N., Agbehadji, I. E., Millham, R., Camacho, D., & Jung, J. J. (2020). Distributed artificial bee colony approach for connected appliances in smart home energy management system. Expert Systems, 37(6), e12521.
- Chaudhry, S. S. (2006). A genetic algorithm approach to solving the anti-covering location problem. Expert Systems, 23(5), 251–257.
- Chavan, A., Karekar, D., Mohandas, M., Manjarekar, R., & Mandhare, S. (2016). Automated question paper generator system using Apriori algorithm and fuzzy logic. International Journal for Innovative Research in Science & Technology, 2(11), 707–710.
- Civicioglu, P. (2012). Transforming geocentric Cartesian coordinates to geodetic coordinates by using differential search algorithm. Computers & Geosciences, 46, 229–247.
- Çomak, E. (2019). A particle swarm optimizer with modified velocity update and adaptive diversity regulation. Expert Systems, 36(1), e12330.
- Darwazeh, A. N., & Branch, R. M. (2015). A revision to the revised Bloom's Taxonomy. 2015 Annual Proceedings–Indianapolis (Vol. 2, pp. 220–225).
- Deb, K., Pratap, A., Agarwal, S., & Meyarivan, T. (2002). A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Transactions on Evolutionary Computation, 6(2), 182–197.
- Emary, E., Zawbaa, H. M., & Hassanien, A. E. (2016). Binary grey wolf optimization approaches for feature selection. Neurocomputing, 172, 371–381.
- Faris, H., Mafarja, M. M., Heidari, A. A., Aljarah, I., Ala'M, A.-Z., Mirjalili, S., & Fujita, H. (2018). An effcient binary salp swarm algorithm with crossover scheme for feature selection problems. Knowledge-Based Systems, 154, 43–67.
- García-Hernández, L., Arauzo-Azofra, A., Salas-Morera, L., Pierreval, H., & Corchado, E. (2015). Facility layout design using a multi-objective interactive genetic algorithm to support the DM. Expert Systems, 32(1), 94–107.
- Gierl, M. J., & Lai, H. (2018). Using automatic item generation to create solutions and rationales for computerized formative testing. Applied Psychological Measurement, 42(1), 42–57.
- Gierl, M. J., Lai, H., & Turner, S. R. (2012). Using automatic item generation to create multiple-choice test items. Medical Education, 46(8), 757–765.
- Greiner, R. (1996). Palo: A probabilistic hill-climbing algorithm. Artificial Intelligence, 84(1–2), 177–208.
- Grun, B., & Zeileis, A. (2009). Automatic generation of exams in R. Journal of Statistical Software, 29(10), 1–14.
- Gursel, A. T., & Kılıç, F. (2021). Optimizing thermal load and power scaling in Er3+-doped ZBLAN fiber lasers. Optical Fiber Technology, 64, 102507. https://doi.org/10.1016/j.yofte.2021.102507
- Hansen, N. (2006). Compilation of results on the 2005 CEC benchmark function set. Online, May.
- Heidari, A. A., Mirjalili, S., Faris, H., Aljarah, I., Mafarja, M., & Chen, H. (2019). Harris hawks optimization: Algorithm and applications. Future Generation Computer Systems, 97, 849–872.
- Holland, J. (1975). Adaptation in natural and artificial systems: An introductory analysis with application to biology, control and artificial intelligence. MIT Press.
- Ince, M., Yigit, T., & Hakan Isik, A. (2020). A novel hybrid fuzzy ahp-ga method for test sheet question selection. International Journal of Information Technology & Decision Making, 19(02), 629–647.
- Karaboga, D. (2005). An idea based on honey bee swarm for numerical optimization (tech. Rep.). Erciyes University, Engineering Faculty.
- Keegan, D. J. (1980). On defining distance education. Distance Education, 1(1), 13–36.
10.1080/0158791800010102 Google Scholar
- Kennedy, J., & Eberhart, R. (1995). Particle swarm optimization. Proceedings of ICNN'95-International Conference on Neural Networks (Vol. 4, pp. 1942–1948).
- Kennedy, J., & Eberhart, R. C. (1997). A discrete binary version of the particle swarm algorithm. 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation (Vol. 5, pp. 4104–4108).
- Kılıç, F., & Gök, M. (2014). A demand based route generation algorithm for public transit network design. Computers & Operations Research, 51, 21–29.
- Kılıç, F., Kaya, Y., & Yildirim, S. (2021). A novel multi population based particle swarm optimization for feature selection. Knowledge-Based Systems, 219, 106894.
- Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983). Optimization by simulated annealing. Science, 220(4598), 671–680.
- Liang, J. J., Qu, B. Y., & Suganthan, P. N. (2013). Problem definitions and evaluation criteria for the cec 2014 special session and competition on single objective real-parameter numerical optimization. Computational Intelligence Laboratory, Zhengzhou University, Zhengzhou China and Technical Report, Nanyang Technological University, Singapore (Vol. 635, p. 490).
- Lin, B., & Miller, D. (2004). Tabu search algorithm for chemical process optimization. Computers & Chemical Engineering, 28(11), 2287–2306.
- Liu, D., Wang, J., & Zheng, L. (2013). Automatic test paper generation based on ant colony algorithm. JSW, 8(10), 2600–2606.
10.4304/jsw.8.10.2600-2606 Google Scholar
- Luantangsrisuk, V., Songmuang, P., & Kongkachandra, R. (2016). Automated test assembly with minimum redundant questions based on bee algorithm. 2016 12th International Conference on Signal-image Technology & Internet-based Systems (SITIS) (pp. 652–656).
- Mafarja, M. M., Eleyan, D., Jaber, I., Hammouri, A., & Mirjalili, S. (2017). Binary dragonfly algorithm for feature selection. 2017 International Conference on New trends in Computing Sciences (ICTCS) (pp. 12–17).
- Ming-zhu, S., Wei-feng, L., & Jing-yi, D. (2013). The research and implementation of technology of generating test paper based on genetic algorithm. In Z. Du (Ed.), Intelligence computation and evolutionary computation (pp. 657–663). Springer Berlin Heidelberg.
10.1007/978-3-642-31656-2_89 Google Scholar
- Mirjalili, S. (2016). Dragonfly algorithm: A new meta-heuristic optimization technique for solving single-objective, discrete, and multi-objective problems. Neural Computing and Applications, 27(4), 1053–1073.
- Mirjalili, S., Gandomi, A. H., Mirjalili, S. Z., Saremi, S., Faris, H., & Mirjalili, S. M. (2017). Salp swarm algorithm: A bio-inspired optimizer for engineering design problems. Advances in Engineering Software, 114, 163–191.
- Mirjalili, S., Mirjalili, S. M., & Lewis, A. (2014). Grey wolf optimizer. Advances in Engineering Software, 69, 46–61.
- Mirjalili, S., Saremi, S., Mirjalili, S. M., & Coelho, L. d. S. (2016). Multi-objective grey wolf optimizer: A novel algorithm for multi-criterion optimization. Expert Systems with Applications, 47, 106–119.
- Naik, K., Sule, S., Jadhav, S., & Pandey, S. (2014). Automatic question paper generation system using randomization algorithm. International Journal of Engineering and Technical Research (IJETR), 2(12), 192–194.
- Nebro, A. J., Durillo, J. J., Garcia-Nieto, J., Coello, C. C., Luna, F., & Alba, E. (2009). Smpso: A new pso-based metaheuristic for multi-objective optimization. 2009 IEEE Symposium on Computational Intelligence in Multi-criteria Decision-making (MCDM) (pp. 66–73).
- Pahnehkolaei, S. M. A., Alfi, A., & Machado, J. T. (2021). Particle swarm optimization algorithm using complex-order derivative concept: A comprehensive study. Applied Soft Computing, 111, 107641.
- Panagant, N., Pholdee, N., Bureerat, S., Kaen, K., Yıldız, A. R., & Sait, S. M. (2020). Seagull optimization algorithm for solving real-world design optimization problems. Materials Testing, 62(6), 640–644.
- Panagant, N., Pholdee, N., Bureerat, S., Yildiz, A. R., & Mirjalili, S. (2021). A comparative study of recent multi-objective metaheuristics for solving constrained truss optimisation problems. Archives of Computational Methods in Engineering, 28, 1–17.
- Rahim, T. N. T. A., Aziz, Z. A., Rauf, R. H. A. & Shamsudin, N. (2017). Automated exam question generator using genetic algorithm. 2017 IEEE Conference on e-learning, e-management and e-services (IC3E) (pp. 12–17). doi: https://doi.org/10.1109/IC3e.2017.8409231
- Sarangkum, R., Wansasueb, K., Panagant, N., Pholdee, N., Bureerat, S., Yildiz, A. R., & Sait, S. M. (2019). Automated design of aircraft fuselage stiffeners using multiobjective evolutionary optimisation. International Journal of Vehicle Design, 80(2-4), 162–175.
- Selim, A., Kamel, S., Alghamdi, A. S., & Jurado, F. (2020). Optimal placement of dgs in distribution system using an improved Harris hawks optimizer based on single-and multi-objective approaches. IEEE Access, 8, 52815–52829.
- Shokri-Ghaleh, H., Alfi, A., Ebadollahi, S., Shahri, A. M., & Ranjbaran, S. (2020). Unequal limit cuckoo optimization algorithm applied for optimal design of nonlinear field calibration problem of a triaxial accelerometer. Measurement, 164, 107963.
- Subramanian, R., Patil, A. P., Ganesan, K., & Akarsh, T. (2019). A semiautomated question paper builder using long short-term memory neural networks. In Emerging research in computing, information, communication and applications (pp. 341–349). Springer.
10.1007/978-981-13-5953-8_29 Google Scholar
- Swart, A. J. (2010). Evaluation of final examination papers in engineering: A case study using Bloom's Taxonomy. IEEE Transactions on Education, 53(2), 257–264.
- Too, J., Abdullah, A. R., & Mohd Saad, N. (2019). A new quadratic binary Harris hawk optimization for feature selection. Electronics, 8(10), 1130.
- Wu, Z., He, T., Mao, C., & Huang, C. (2020). Exam paper generation based on performance prediction of student group. Information Sciences, 532, 72–90.
- Yang, J. (2015). Design on generating test paper based on simulated annealing algorithm. 2nd International Conference on Civil, Materials and Environmental Sciences (pp. 689–693).
- Yang, X.-S., & Deb, S. (2009). Cuckoo search via lévy flights. 2009 World Congress on Nature & Biologically Inspired Computing (NABIC) (pp. 210–214).
- Yildirim, M. (2010). A genetic algorithm for generating test from a question bank. Computer Applications in Engineering Education, 18(2), 298–305.
- Yıldız, A. R., & Erdas, M. U. (2021). A new hybrid taguchi-salp swarm optimization algorithm for the robust design of real-world engineering problems. Materials Testing, 63(2), 157–162.
- Zhang, C., & Zhang, J. (2015). Research on test paper auto-generating based on improved particle swarm optimization. 2015 Seventh International Symposium on Parallel Architectures, Algorithms and Programming (PAAP) (pp. 92–96).
- Zomorodi-moghadam, M., Abdar, M., Davarzani, Z., Zhou, X., Pławiak, P., & Acharya, U. R. (2021). Hybrid particle swarm optimization for rule discovery in the diagnosis of coronary artery disease. Expert Systems, 38(1), e12485.
