e-ISSN: 2618-6586 Open Access · Peer-Reviewed JETOL on DergiPark
JETOL — Journal of Educational Technology & Online Learning
Publisher
Gürhan Durak
Publication Model
Periodical (January – May – September)
Status
Open for Submissions

Comprehensive AI assessment framework: Enhancing educational evaluation with ethical AI integration

Research Article

  • Selçuk Kılınç Orta Doğu Teknik Üniversitesi, Fen Bilimleri Enstitüsü, Fen Bilimleri Eğitimi (dr) 0000-0001-8846-7243
Download PDF DOI: 10.31681/jetol.1492695
Pages
521–540
Publication Date
December 31, 2024
Submission Date
May 31, 2024
Acceptance Date
October 18, 2024
Subjects
Instructional Technologies

Abstract

The integration of generative artificial intelligence (GenAI) tools into education has been a game-changer for teaching and assessment practices, bringing new opportunities, but also novel challenges which need to be dealt with. This paper presents the Comprehensive AI Assessment Framework (CAIAF), an evolved version of the AI Assessment Scale (AIAS) by Perkins, Furze, Roe, and MacVaugh, targeted toward the ethical integration of AI into educational assessments. This is where the CAIAF differs, as it incorporates stringent ethical guidelines, with clear distinctions based on educational levels, and advanced AI capabilities of real-time interactions and personalized assistance. The framework developed herein has a very intuitive use, mainly through the use of a color gradient that enhances the user-friendliness of the framework. Methodologically, the framework has been developed through the huge support of a thorough literature review and practical insight into the topic, becoming a dynamic tool to be used in different educational settings. The framework will ensure better learning outcomes, uphold academic integrity, and promote responsible use of AI, hence the need for this framework in modern educational practice.

Keywords

  • Comprehensive AI Framework
  • AI Assessment
  • AI Integration
  • AI Ethics
  • AI in Education
References (81)
  1. Adıgüzel, T., Kaya, M., & Cansu, F. (2023). Revolutionizing education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 15(3), ep429. https://doi.org/10.30935/cedtech/13152
  2. Badeni, B., & Saparahayuningsih, S. (2019). Who is responsible for the child’s moral character education? Education Quarterly Reviews, 2(1), 23-32. https://doi.org/10.31014/aior.1993.02.01.35
  3. Borenstein, J., & Howard, A. (2021). Emerging challenges in AI and the need for AI ethics education. AI Ethics, 1, 61-65. https://doi.org/10.1007/s43681-020-00002-7
  4. Boscardin, C. (2024). ChatGPT and generative artificial intelligence for medical education: Potential impact and opportunity. Academic Medicine, 99(1), 22-27. https://doi.org/10.1097/acm.0000000000005439
  5. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., … & Zhang, Y. (2023). Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprinarXiv:2303.12712.
  6. Chakraborty, S., Bedi, A. S., Zhu, S., An, B., Manocha, D., & Huang, F. (2023). On the possibilities of ai-generated text detection. arXiv preprint arXiv:2304.04736. https://doi.org/10.48550/arXiv.2304.04736
  7. Chan, C. K. Y., & Tsi, L. H. (2023). The AI revolution in education: Will AI replace or assist teachers in higher education? arXiv preprint arXiv:2305.01185.
  8. Chang, D. H., Lin, M. P. C., Hajian, S., & Wang, Q. Q. (2023). Educational design principles of using AI chatbot that supports self-regulated learning in education: Goal setting, feedback, and personalization. Sustainability, 15(17), 12921. https://doi.org/10.3390/su151712921
  9. De Laat, P. (2021). Companies committed to responsible AI: From principles towards implementation and regulation? Philosophy & Technology, 34(4), 1135-1193. https://doi.org/10.1007/s13347-021-00474-3
  10. Dickey, E., & Bejarano, A. (2023). A model for integrating generative AI into course content development. arXiv preprint arXiv:2308.12276.
  11. Elliot, A. J., & Maier, M. A. (2014). Color psychology: Effects of perceiving color on psychological functioning in humans. Annual Review of Psychology, 65, 95-120. https://doi.org/10.1146/annurev-psych-010213-115035
  12. Foray, D., & Raffo, J. (2012). Business-driven innovation: Is it making a difference in education? An analysis of educational patents. http://dx.doi.org/10.1787/5k91dl7pc835-en
  13. Frankel, F., & DePace, A. H. (2012). Visual strategies: A practical guide to graphics for scientists & engineers. Yale University Press.
  14. Gampala, S., Vankeshwaram, V., & Gadula, S. S. P. (2020). Is artificial intelligence the new friend for radiologists? A review article. Cureus, 12(10). https://doi.org/10.7759/cureus.11137
  15. Gao, X. (2024). Language education in a brave new world: A dialectical imagination. Modern Language Journal, 108(2), 556-562. https://doi.org/10.1111/modl.12930
  16. García-Martínez, I., Fernández-Batanero, J. M., Fernández-Cerero, J., & León, S. P. (2023). Analysing the impact of artificial intelligence and computational sciences on student performance: Systematic review and meta-analysis. Journal of New Approaches in Educational Research, 12(1), 171-197. https://doi.org/10.7821/naer.2023.1.1240
  17. Garg, S., & Sharma, S. (2020). Impact of artificial intelligence in special need education to promote inclusive pedagogy. International Journal of Information and Education Technology, 10(7), 523-527. https://doi.org/10.18178/ijiet.2020.10.7.1418
  18. Gillani, N., Eynon, R., Chiabaut, C., & Finkel, K. (2023). Unpacking the “Black Box” of AI in education. Educational Technology & Society, 26(1), 99-111. https://doi.org/10.30191/ETS.202301_26(1).0008
  19. Green, C., Mynhier, L., Banfill, J., Edwards, P., Kim, J., & Desjardins, R. (2020). Preparing education for the crises of tomorrow: A framework for adaptability. International Review of Education, 66, 857-879. https://doi.org/10.1007/s11159-020-09878-3
  20. Hamiti, M., Reka, B., & Baloghová, A. (2014). Ethical use of information technology in high education. Procedia-Social and Behavioral Sciences, 116, 4411-4415. https://doi.org/10.1016/j.sbspro.2014.01.957
  21. Holmes, W., Porayska-Pomsta, K., Holstein, K., Sutherland, E., Baker, T., Shum, S. B., … & Koedinger, K. R. (2022). Ethics of AI in education: Towards a community-wide framework. International Journal of Artificial Intelligence in Education, 1-23. https://doi.org/10.1007/s40593-021-00239-1
  22. Hong, Y., Nguyen, A., Dang, B., & Nguyen, B. P. T. (2022, July). Data ethics framework for artificial intelligence in education (AIED). In 2022 International Conference on Advanced Learning Technologies (ICALT) (pp. 297-301). IEEE. https://doi.org/10.1109/ICALT55010.2022.00095
  23. Jones, S. (2016). Doing the right thing: Computer ethics pedagogy revisited. Journal of Information, Communication and Ethics in Society, 14(1), 33-48. https://doi.org/10.1108/JICES-07-2014-0033
  24. Kamalov, F., & Gurrib, I. (2023). A new era of artificial intelligence in education: A multifaceted revolution. arXiv preprint arXiv:2305.18303.
  25. Kitsios, F., & Kamariotou, M. (2021). Artificial intelligence and business strategy towards digital transformation: A research agenda. Sustainability, 13(4), 2025. https://doi.org/10.3390/su13042025
  26. Kılınç, S. (2023). Embracing the future of distance science education: Opportunities and challenges of ChatGPT integration. Asian Journal of Distance Education, 18(1), 205-237. https://doi.org/10.5281/zenodo.7857396
  27. Klimova, B., Pikhart, M., & Kacetl, J. (2023). Ethical issues of the use of AI-driven mobile apps for education. Frontiers in Public Health. https://doi.org/10.3389/fpubh.2022.1118116
  28. Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of intelligent tutoring systems: a meta-analytic review. Review of educational research, 86(1), 42-78. https://doi.org/10.3102/0034654315581420
  29. Lam, T., Cheung, M., Munro, Y., Lim, K., Shung, D., & Sung, J. (2022). Randomized controlled trials of artificial intelligence in clinical practice: Systematic review. Journal of Medical Internet Research, 24(8), e37188. https://doi.org/10.2196/37188
  30. Lane, S. H., Haley, T., & Brackney, D. E. (2024). Tool or tyrant: Guiding and guarding generative artificial intelligence use in nursing education. Creative Nursing, 30(2), 125-132. https://doi.org/10.1177/10784535241247094
  31. Leimanis, A. (2020, February). Self-imposed ethical guidelines for AI in education. In International Scientific Conference “SOCIETY, INTEGRATION, EDUCATION-SIE2020”.
  32. Luckin, R., & Holmes, W. (2016). Intelligence unleashed: An argument for AI in education.
  33. Ma, X., & Jiang, C. (2023). On the ethical risks of artificial intelligence applications in education and its avoidance strategies. Journal of Education, Humanities and Social Sciences, 14, 354-359. https://doi.org/10.54097/ehss.v14i.8868
  34. Maghsudi, S., Lan, A., Xu, J., & van Der Schaar, M. (2021). Personalized education in the artificial intelligence era: What to expect next. IEEE Signal Processing Magazine, 38(3), 37-50. https://doi.org/10.1109/MSP.2021.3055032
  35. Mahligawati, F., Allanas, E., Butarbutar, M. H., & Nordin, N. A. N. (2023, September). Artificial intelligence in physics education: A comprehensive literature review. In Journal of Physics: Conference Series (Vol. 2596, No. 1, p. 012080). IOP Publishing. https://doi.org/10.1088/1742-6596/2596/1/012080
  36. Mâță, L. (2022). Ethical rules of online communication between university teachers and students. In: Mâță, L. (Ed.) Ethical use of information technology in higher education. EAI/Springer Innovations in Communication and Computing. Springer, Singapore. https://doi.org/10.1007/978-981-16-1951-9_7
  37. Mayer, J. (2009). The growing interdependence between financial and commodity markets (No. 195). United Nations Conference on Trade and Development.
  38. Mello, R. F., Freitas, E., Pereira, F. D., Cabral, L., Tedesco, P., & Ramalho, G. (2023). Education in the age of generative AI: Context and recent developments. arXiv preprint arXiv:2309.12332.
  39. Michaeli, T., Romeike, R., & Seegerer, S. (2022, August). What students can learn about artificial intelligence–recommendations for K-12 computing education. In IFIP World Conference on Computers in Education (pp. 196-208). Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-43393-1_19
  40. Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers college record, 108(6), 1017-1054. https://doi.org/10.1111/j.1467-9620.2006.00684.x
  41. Mollick, E. R., & Mollick, L. (2023a). Using AI to implement effective teaching strategies in classrooms: Five strategies, including prompts. Including Prompts (March 17, 2023). http://dx.doi.org/10.2139/ssrn.4391243
  42. Mollick, E., & Mollick, L. (2023b). Assigning AI: Seven approaches for students, with prompts. arXiv preprint arXiv:2306.10052.
  43. Morley, J., Elhalal, A., Garcia, F., Kinsey, L., Mökander, J., & Floridi, L. (2021). Ethics as a service: A pragmatic operationalisation of AI ethics. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3784238
  44. Network, P. O. D. (2001). Ethical guidelines for educational developers. To Improve the Academy, 19.
  45. Ng, P. T. (2009). Innovation in education: Some observations and questions. International Journal of Innovation in Education, 1(1), 8-11. https://doi.org/10.1504/IJIIE.2009.0301
  46. Ng, D. T. K., Leung, J. K. L., Su, J., Ng, R. C. W., & Chu, S. K. W. (2023). Teachers’ AI digital competencies and twenty-first century skills in the post-pandemic world. Educational technology research and development, 71(1), 137-161. https://doi.org/10.1007/s11423-023-10203-6
  47. Nguyen, A., Ngo, H. N., Hong, Y., Dang, B., & Nguyen, B. P. T. (2023). Ethical principles for artificial intelligence in education. Education and Information Technologies, 28(4), 4221-4241. https://doi.org/10.1007/s10639-022-11316-w
  48. Ogunleye, B., Zakariyyah, K. I., Ajao, O., Olayinka, O., & Sharma, H. (2024). Higher education assessment practice in the era of generative AI tools. arXiv preprint arXiv:2404.01036.
  49. Olga, A., Saini, A., Zapata, G., Searsmith, D., Cope, B., Kalantzis, M., … & Kastania, N. P. (2023). Generative AI: Implications and applications for education. arXiv preprint arXiv:2305.07605.
  50. Pack, A., & Maloney, J. (2024). Using artificial intelligence in TESOL: Some ethical and pedagogical considerations. TESOL Quarterly, 58(2), 1007-1018. https://doi.org/10.1002/tesq.3320
  51. Park, W., & Kwon, H. (2024). Implementing artificial intelligence education for middle school technology education in Republic of Korea. International Journal of Technology and Design Education, 34(1), 109-135. https://doi.org/10.1007/s10798-023-09812-2
  52. Pasricha, S. (2023, June). Ethics in computing education: Challenges and experience with embedded ethics. In Proceedings of the Great Lakes Symposium on VLSI 2023 (pp. 653-658). https://doi.org/10.1145/3583781.3590240
  53. Perkins, M., Furze, L., Roe, J., & MacVaugh, J. (2024). The artificial intelligence assessment scale (AIAS): A framework for ethical integration of generative AI in educational assessment. Journal of University Teaching and Learning Practice, 21(06). https://doi.org/10.53761/q3azde36
  54. Perkins, M., Roe, J., Vu, B. H., Postma, D., Hickerson, D., McGaughran, J., & Khuat, H. Q. (2024). GenAI detection tools, adversarial techniques and implications for inclusivity in higher education. arXiv preprint arXiv:2403.19148.
  55. Poza-Lujan, J. L., Calafate, C. T., Posadas-Yagüe, J. L., & Cano, J. C. (2015). Assessing the impact of continuous evaluation strategies: Tradeoff between student performance and instructor effort. IEEE Transactions on Education, 59(1), 17-23. https://doi.org/10.1109/TE.2015.2418740
  56. Pushpa, S. (2012). Ethical leadership: Need for business ethics education. International Journal of Advances in Management and Economics, 1(1), 14-19.
  57. Puyo, J. G. B. (2021). A value and character educational model: Repercussions for students, teachers, and families. Journal of Culture and Values in Education, 4(1), 100-115. https://doi.org/10.46303/jcve.2020.7
  58. Rafikov, I., Akhmetova, E., & Yapar, O. E. (2021). Prospects of morality-based education in the 21st century. Journal of Islamic Thought and Civilization, 11(1), 1-21. https://doi.org/10.32350/jitc.111.01
  59. Sajja, R., Sermet, Y., Cikmaz, M., Cwiertny, D., & Demir, I. (2023). Artificial intelligence-enabled intelligent assistant for personalized and adaptive learning in higher education. arXiv preprint arXiv:2309.10892.
  60. Selwyn, N. (2013). Distrusting educational technology: Critical questions for changing times. Routledge.
  61. Shih, P. K., Lin, C. H., Wu, L. Y., & Yu, C. C. (2021). Learning ethics in AI—teaching non-engineering undergraduates through situated learning. Sustainability, 13(7), 3718. https://doi.org/10.3390/su13073718
  62. Simms, R. C. (2024). Work with ChatGPT, not against: 3 teaching strategies that harness the power of artificial intelligence. Nurse Educator, 49(3), 158-161. https://doi.org/10.1097/nne.0000000000001634
  63. Singh, S., & Riedel, S. (2016). Creating interactive and visual educational resources for AI. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). https://doi.org/10.1609/aaai.v30i1.9851
  64. Singh, S. V., & Hiran, K. K. (2022). The impact of AI on teaching and learning in higher education technology. Journal of Higher Education Theory and Practice, 22(13).
  65. Stobart, G. (2004). Developing and improving assessment instruments.
  66. Sullivan, M., Kelly, A., & McLaughlan, P. (2023). ChatGPT in higher education: Considerations for academic integrity and student learning. https://doi.org/10.37074/jalt.2023.6.1.17
  67. Swiecki, Z., Khosravi, H., Chen, G., Martinez-Maldonado, R., Lodge, J. M., Milligan, S., … & Gašević, D. (2022). Assessment in the age of artificial intelligence. Computers and Education: Artificial Intelligence, 3, 100075. https://doi.org/10.1016/j.caeai.2022.100075
  68. Tong, R., Li, H., Liang, J., & Wen, Q. (2024). Developing and Deploying Industry Standards for Artificial Intelligence in Education (AIED): Challenges, Strategies, and Future Directions. arXiv preprint arXiv:2403.14689.
  69. Volante, L., DeLuca, C., & Klinger, D. A. (2023). Leveraging AI to enhance learning. Phi Delta Kappan, 105(1), 40-45. https://doi.org/10.1177/00317217231197475
  70. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes (Vol. 86). Harvard university press.
  71. Wang, Z., Chen, A., Tao, K., Han, Y., & Li, J. (2024). MatGPT: A vane of materials informatics from past, present, to future. Advanced Materials, 36(6). https://doi.org/10.1002/adma.202306733
  72. Wang, Z., & Zhai, J. (2019, October). Ethical challenges faced by students in the educational environment of artificial intelligence. In 2019 International Conference on Advanced Education Research and Modern Teaching (AERMT 2019) (pp. 1-3). Atlantis Press. https://doi.org/10.2991/aermt-19.2019.1
  73. Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., … & Waddington, L. (2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity, 19(1), 26. https://doi.org/10.1007/s40979-023-00146-z
  74. Xu, X., Zhang, J., Zhu, Q., & Xia, T. (2023). The influences of gradient color on the weight perception and stability perception: A preliminary study. i-Perception, 14(4), 20416695231197797. https://doi.org/10.1177/20416695231197797
  75. Yen, W. M., Lall, V. F., & Monfils, L. (2012). Evaluating academic progress without a vertical scale. ETS Research Report Series, 2012(1), i-55. https://doi.org/10.1002/j.2333-8504.2012.tb02289.x
  76. Yeo, K. (2023). Artificial intelligence in cardiology: Did it take off? Russian Journal for Personalized Medicine, 2(6), 16-22. https://doi.org/10.18705/2782-3806-2022-2-6-16-22
  77. Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education–where are the educators?. International Journal of Educational Technology in Higher Education, 16(1), 1-27. https://doi.org/10.1186/s41239-019-0171-0
  78. Zeileis, A., Hornik, K., & Murrell, P. (2009). Escaping RGBland: Selecting colors for statistical graphics. Computational Statistics & Data Analysis, 53(9), 3259-3270. https://doi.org/10.1016/j.csda.2008.11.033
  79. Zhang, C., Zhang, C., Li, C., Qiao, Y., Zheng, S., Dam, S. K., … & Hong, C. S. (2023). One small step for generative AI, one giant leap for AGI: A complete survey on ChatGPT in AIGC era. arXiv preprint arXiv:2304.06488.
  80. Zhang, K., & Aslan, A. B. (2021). AI technologies for education: Recent research & future directions. Computers and Education: Artificial Intelligence, 2. https://doi.org/10.1016/j.caeai.2021.100025
  81. Zhou, X., Van Brummelen, J., & Lin, P. (2020). Designing AI learning experiences for K-12: Emerging works, future opportunities and a design framework. arXiv preprint arXiv:2009.10228.

This article is published under the CC BY 4.0 license.