A Framework for Evaluating Distance Learning of Environmental Science in Higher Education using Multi-Criteria Group Decision Making
Abstract
Purpose: Due to Covid-19, big changes took place in Universities around the world. Universities were asked in March 2020 within a short while to provide the whole of the available lessons using e-Learning methods. Since the health crisis continues, e-learning has expanded on a variety of contexts and simultaneously has created an urgent need for designing, developing, and implementing a valid and reliable distance learning procedure. The validity and efficiency of the aforementioned procedure is a major issue that has to be tested at the end of the semester. Therefore, developing a valid framework to evaluate the e-learning process has become more important now than in the past due to the ongoing pandemic.
Design/methodology/approach: The evaluation of the educational process is a multi-criteria problem that is based on the points of view of both instructors and students. In order to solve this multi-criteria problem of e-learning evaluation, a new framework for evaluating e-Learning in Higher Education has been developed. This framework uses group decision making with multiple criteria and is called ENVEL. This paper defines the set of evaluation criteria and uses the Fuzzy Analytic Hierarchy Process to prioritize criteria and help the decision-makers draw conclusions on the effectiveness and success of e-learning.
Findings: The framework takes into account heterogeneous groups of students and professors, makes different calculations for these groups, and can extract useful conclusions by comparing the results of the different groups. The framework has been applied in the Department of Environmental Science at the Ionian University and conclusions have been made on its effectiveness and usage.
Originality: Trying to focus on the evaluation of e-learning in a whole study program in Higher Education, and not only on single courses, the paper describes a novel framework for e-learning evaluation using multi-criteria decision making with heterogeneous groups of users. This framework provides a formal way of combining different aspects of the evaluation of e-learning and collect summative results.
Full Text:
PDFReferences
Abdalla, I. (2007). Evaluating effectiveness of e-blackboard system using TAM framework: A structural analysis approach. AACE Journal, 15(3), 279-287.
Abuhlfaia K., de Quincey E. (2019). Evaluating the Usability of an e-Learning Platform within Higher Education from a Student Perspective. ICEEL 2019: Proceedings of the 2019 3rd International Conference on Education and E-LearningPages 1–7https://doi.org/10.1145/3371647.3371661
Aguti, B., Wills, G.B., Walters, R.J. (2014). An Evaluation of the Factors that Impact on the Effectiveness of Blended E-Learning within Universities. International Conference on Information Society, 117-121.
Alavi, M., & Leidner, D. E. (2001). Technology-mediated learning: A call for greater depth and breadth of research. Information Systems Research, 12(1), 1–10.
Al-Fadhli, S. & Khalfan, A. (2009). Developing critical thinking in e-learning environment: Kuwait University as a case study. Assessment & Evaluation in Higher Education, 34(5), 529-536, DOI: 10.1080/02602930802117032
Alqahtani A.Y., Rajkhan A.A. (2020) E-Learning Critical Success Factors during the COVID-19 Pandemic: A Comprehensive Analysis of E-Learning Managerial Perspectives. Educational Sciences, vol. 10, 216
Altun Turker Y., Baynal K., Turker T. (2019) The Evaluation of Learning Management Systems by Using Fuzzy AHP, Fuzzy TOPSIS and an Integrated Method: A Case Study, Turkish Online Journal of Distance Education, 20(2):195-218, DOI: 10.17718/tojde.557864
Anderson, T. (2001). The hidden curriculum in distance education. An update view, Change 33, 28-35.
Anggrainingsih, R. , Umam, M. Z. , Setiadi, H. (2018) Determining e-learning success factor in higher education based on user perspective using Fuzzy AHP. MATEC Web of Conferences 154, 03011, https://doi.org/10.1051/matecconf/201815403011
Anis A. & Islam R. (2015) The application of analytic hierarchy process in higher-learning institutions: a literature review. J. International Business and Entrepreneurship Development, Vol. 8, No. 2, 2015
Arbaugh, J. B. (2004). Learning to learn online: A study of perceptual changes between multiple online course experiences, The Internet and Higher Education, 7, 169-182
Arbaugh, J. B. (2008). Does the community of inquiry framework predict outcomes in online MBA courses?. International Review of Research in Open and Distance Learning, 9(2).
Barteit S, Guzek D, Jahn A, Bärnighausen T, Jorge MM, Neuhann F. Evaluation of e-learning for medical education in low- and middle-income countries: A systematic review. Comput Educ. 2020 Feb;145:103726. doi: 10.1016/j.compedu.2019.103726. PMID: 32565611; PMCID: PMC7291921.
Benbunan-Fich, R. & Hiltz, S. R. (2003) Mediators of the effectiveness of online courses, IEEE Transactions on Professional Communication 46 (2003), 298 312.
Bhadauria R. (2016) E-Learning – A boon for Indian Higher Education System. International Journal of Engineering Technology, Management and Applied Sciences. 4(2), 122-128, ISSN 2349-4476
Buckley, J.J. (1985). Fuzzy hierarchical analysis. Fuzzy Sets and Systems, 17(3), 233-247.
Yakup Çelikbilek & Ayşe Nur Adıgüzel Tüylü (2019): Prioritizing the components of e-learning systems by using fuzzy DEMATEL and ANP, Interactive Learning Environments, DOI: 10.1080/10494820.2019.1655065
Chang, C. W., Wu, C. R., & Lin, H. L. (2008) Integrating fuzzy theory and hierarchy concepts to evaluate software quality, Software Quality Journal, 16(2), 263–76.
Chen, J.-F., Hsieh, H.-N., & Do, Q. H. (2015) Evaluating teaching performance based on fuzzy AHP and comprehensive evaluation approach, Applied Soft Computing, 28, 100–108.
Chou, S.Y., Chang, Y.H., Shen, C.Y. (2008) A fuzzy simple additive weighting system under group decision-making for facility location selection with objective/subjective attributes. European Journal of Operational Research 189 (2008) 132–145.
Chou, S.-W., & Liu, C.-H. (2005) Learning effectiveness in a web-based virtual learning environment: A learner control perspective, Journal of Computer Assistive Learning, 21, 65-76
Clark, L. J. (2001). Web-based teaching: A new educational paradigm, Intercom 48, 20-23.
Elkaseh A.M., Wong K.W. and Fung C.C.(2016) Perceived ease of use and perceived usefulness of social media for e-learning in Libyan higher education: a structural equation modeling analysis. International Journal of Information and Education Technology 6(3), 192-199.
Erensal, Y. C., Öncan, T., & Demircan, M. L. (2006). Determining key capabilities in technology management using fuzzy analytic hierarchy process: A case study of Turkey, Information Sciences, 176(18), 2755–70.
Frehywot, S., Vovides, Y., Talib, Z. et al. E-learning in medical education in resource constrained low- and middle-income countries. Hum Resour Health 11, 4 (2013). https://doi.org/10.1186/1478-4491-11-4
Gupta, S., & Bostrom, R.P. (2009). Technology-Mediated Learning: A Comprehensive Theoretical Model. Journal of the Association for Information Systems, 10(9), 686-714.
Harasim, L., Hiltz, S. R., Teles, L., & Turoff, M. (1995) Learning networks: A field guide to teaching and learning online. The MIT Press, Cambridge, MA.
Harrati, N., Bouchrika, I., Tari, A., and Ladjailia, A. 2016. Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis. Comput Hum Behav, 61, 463--471. DOI= https://doi.org/10.1016/j.chb.2016.03.051.
Hartman, K., Neuwirth, C. M., Kiesler, S., Sproull, L., Cochran, C., Palmquist, M., & Zubrow, D. (1995) Patterns of social interaction and learning to write: Some effects of network technologies. In: Z. L. Berge and M. P. Collins, Editors. Computer mediated communication and the online classroom, Vol. 2. Hampton Press, Inc, Cresskill, NJ, 1995, pp 4778.
Haverila, M., & Barkhi, R. (2009). The Influence of Experience, Ability and Interest on eLearning Effectiveness. European Journal of Open, Distance and E-Learning, https://www.eurodl.org/materials/contrib/2009/Haverila_Barkhi.pdf
Hrastinski, S. (2008) Asynchronous and synchronous e-learning. Educause Quarterly, 31(4), 51-55.
Hwang, C.L., Yoon, K. Multiple Attribute Decision Making: Methods and Applications. Lecture Notes in Economics and Mathematical Systems, Vol. 186, 1981.
Islam, A. Md., Rahim, N.A.N., Liang, T.C., & Momtaz, H. (2011) Effect of Demographic Factors on E-Learning Effectiveness in A Higher Learning Institution in Malaysia. International Education Studies. 4(1), 112-121.
Islas, E., Perez, M., Rodriguez, G., Paredes, I., Avila, I., & Mendoza, M. (2007), E-learning tools evaluation and roadmap development for an electrical utility, Journal of Theoretical and Applied Electronic Commerce Research, 2(1), 63-75.
Jasim, M. H., Maznah M. K., Nizal M. S. I. (2018) Evaluation of E-Learning Approaches Using AHPTOPSIS Technique. Journal of Telecommunication, Electronic and Computer Engineering, 10 (1-10). pp. 7-10. ISSN 2180-1843.
Jeong J. & González-Gómez D. (2020). Assessment of sustainability science education criteria in online-learning through fuzzy-operational and multi-decision analysis and professional survey, Heliyon, Heliyon, Volume 6, Issue 8, August 2020, e04706, https://doi.org/10.1016/j.heliyon.2020.e04706
Liaw, S.S. (2008). Investigating students’ perceived satisfaction, behavioral intention, and effectiveness of e-learning: A case study of the Blackboard system. Computers & Education, 51, 864–873
Liaw, S. S., Huang, H. M., & Chen, G. D., (2007). Surveying instructor and learner attitudes toward e-learning, Computers & Education. https://doi.org/10.1016/j.compedu.2006.01.001
Lin, H.-F. (2010) An application of fuzzy AHP for evaluating course website quality. Computers & Education 54 (2010) 877–888
Linjawi, AI, Alfadda, LS. (2018) Students’ perception, attitudes, and readiness toward online learning in dental education in Saudi Arabia: a cohort study. Advances in Medical Education and Practice Med Educ Pract. 9:855-863. https://doi.org/10.2147/AMEP.S175395
Mahdavi, I., Fazlollahtabar, H., Heidarzade, A., Mahdavi-Amiri N. and Rooshan, Y.I. (2008). A Heuristic Methodology for Multi-Criteria Evaluation of Web-Based E-Learning Systems Based on User Satisfaction. Journal of Applied Sciences, 8: 4603-4609. DOI: 10.3923/jas.2008.4603.4609
Marks, R. B., Sibley, S. D., & Arbaugh, J. B. (2005). A structural equation model of predictor for effective online learning, Journal of Management Education, 29, 531-563.
Martinez-Caro, E. (2011). Factors Affecting Effectiveness in E-Learning: An Analysis in Production Management Courses. Computer Applications in Engineering Education, 19(3), 572 – 581. DOI: 10.1002/cae.20337
Martín-Lara, M.Á.; Rico, N. Education for Sustainable Energy: Comparison of Different Types of E-Learning Activities. Energies 2020, 13, 4022.
Mercader, C., Gairín, J. University teachers’ perception of barriers to the use of digital technologies: The importance of the academic discipline. Int. J. Educ. Technol. High. Educ. 2020, 17, 4.
Mohamadali, N.A., & Garibaldi, J. (2011). Comparing user acceptance factors between research software and medical software using AHP and Fuzzy AHP. In: The 11th Workshop on Computational Intelligence, 7 - 9 September 2011, Kilburn Building.
Mulubrhan, F., Akmar, A., Mokhtar, A. & Muhammad, M. (2014). Comparative Analysis between Fuzzy and Traditional Analytical Hierarchy Process. MATEC Web of Conferences 13.
Nagpal, R., Mehrotra, D. Bhatia, P. K., & Sharma, A. (2015) FAHP approach to rank educational websites on usability, International Journal of Computing and Digital Systems, 4(4), 251–260.
Naveed, QN, Qureshi, MRN, Tairan, N, Mohammad, A, Shaikh, A, Alsayed, AO, et al. (2020) Evaluating critical success factors in implementing E-learning system using multi-criteria decision making. PLoS ONE 15(5): e0231465. https://doi. org/10.1371/journal.pone.0231465
Njenga, J.K. E-Learning in Higher Education. In Encyclopedia of International Higher Education Systems and Institutions; Springer: Maine, The Netherlands, 2017; pp. 1–6. [CrossRef]
Noesgaard, S. S., & Ørngreen, R. (2015) The Effectiveness of E-Learning: An Explorative and Integrative Review of the Definitions, Methodologies and Factors that Promote e-Learning Effectiveness. The Electronic Journal of eLearning, 13(4), 278-290.
Opricovic, S. (1998). Multicriteria optimization of civil engineering systems. Belgrade: Faculty of Civil Engineering
Otto, D., Becker, S. E-Learning and Sustainable Development. In Encyclopedia of Sustainability in Higher Education; Springer: Cham, Switzerland, 2018; p. 8.
Owston, R. (1997) The World Wide Web: A technology to enhance teaching and learning?, Educational Researcher, 26(2).
Ramanayaka, K. H., Chen, X., & Shi, B. (2019). UNSCALE: A Fuzzy-based Multi-criteria Usability Evaluation Framework for Measuring and Evaluating Library Websites, IETE Technical Review, 36(4), 412-431, DOI: 10.1080/02564602.2018.1498032
Reiser, R. (2001). A History of Instructional Design & Technology: Part 1: A History of Instructional Media. Educational Research & Development, 49 (1), 53-64.
Richmond, V. P., McCroskey, J. C., Kearney, P. & Plax, T. G. (1987) Power in classroom VII: Linking behaviour alteration techniques to cognitive learning, Communication Education, 36, 1-12.
Saaty, T. (1980). The analytic hierarchy process. New York, McGraw-Hill.
Sadeghi, M., ali Rashidzadeh, M., & ali Soukhakian, M. (2012). Using analytic network process in a group decision making for supplier selection, Informatica, 23(4), 621–43.
Salter, S.M., Karia, A., Sanfilippo, F.M., Clifford, R.M. (2014). Effectiveness of E-learning in Pharmacy Education. American Journal of Pharmaceutical Education, 78(4), Article 83.
Schieffer, L. The Benefits and Barriers of Virtual Collaboration Among Online Adjuncts. J. Instr. Res. 2016, 5, 109–125.
Sheikh, T. H., Gupta R. (2018). E-Learning for Higher Education. A Case Study. Journal of Advanced Research in Dynamical & Control Systems, 10(4), ISSN 1943-023X
Siemens, G., Gasevic D. & Dawson S. (2015). Preparing for the digital university: A review of the history and current state of distance, blended and online learning. Athabasca: Athabasca University. DOI: 10.13140/RG.2.1.3515.8483
Sivaji, A., Abdullah, M. R., Downe, A. G. & Ahmad, W. F. W. (2013) Hybrid Usability Methodology: Integrating Heuristic Evaluation with Laboratory Testing across the Software Development Lifecycle, 10th International Conference on Information Technology: New Generations, 375–83.
Stecyk A. (2019) Application of PROMETHEE II method for e-learning course selection e-mentor, 1(78), 39-45. DOI: 10.15219/em80.1398
Stöhr C., Demaziere C., Adawi T. (2020) The polarizing effect of the online flipped classroom. Computers & Education, 147, 103789
Su, B., Bonk, C. J., Magjuka, R.J., Liu, X. & Lee, S. (2005) The importance on Interaction in web-based education: A program level case study of online MBA courses, Journal of Interactive Online Learning, 4, 119.
Sulčič, V., & Lesjak, D. (2009). E-Learning and Study Effectiveness, Journal of Computer Information Systems, 49(3), 40-47.
Tai D.W.S., Chen JL., Wang R. (2011) Research on Fuzzy Linguistic Evaluation in e-Learning Using AHP and TOPSIS Based on Web Resources. In: Lin S., Huang X. (eds) Advances in Computer Science, Environment, Ecoinformatics, and Education. CSEE 2011. Communications in Computer and Information Science, vol 217. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-23339-5_69
Tiwari, N. (2006). Using the Analytic Hierarchy Process (AHP) to identify Performance Scenarios for Enterprise Application, Computer Measurement Group, Measure It, 4, 3.
Tseng, M.-L., & Lin, R.-J., Chen, H.-P. (2011). Evaluating the effectiveness of e-learning system in uncertainty. Industrial Management & Data Systems, 111(6), 869-889.
Vincke, P.: Multicriteria Decision-Aid. Wiley, Chichester (1992)
I. Vinogradova, R. Kliukas (2015) Methodology for Evaluating the Quality of Distance Learning Courses in Consecutive Stages, Procedia - Social and Behavioral Sciences,
Volume 191, Pages 1583-1589, https://doi.org/10.1016/j.sbspro.2015.04.364.
Wu, W. & Hwang, L.-Y. (2010). The effectiveness of e-learning for blended courses in colleges: A multi-level empirical study. International Journal of Electronic Business Management, 8(4), 312-322.
Zopounidis, C. (2009). Knowledge-based multi-criteria decision support. European journal of operational research, 195, 827-828.
DOI: https://doi.org/10.31449/inf.v47i4.3401
This work is licensed under a Creative Commons Attribution 3.0 License.