The Role of AI Literacy and Generative AI Reliance in Decision Making Tasks: Case Studies Using Business   Simulation Games

Phaninthorn Swanyawatthaga; Thanachart Ritbumroong

Authors

Phaninthorn Swanyawatthaga Management of Analytics and Data Technologies, Graduate School of Applied Statistics, National Institute of Development Administration
Thanachart Ritbumroong Management of Analytics and Data Technologies, Graduate School of Applied Statistics, National Institute of Development Administration

Keywords:

Generative AI, AI Literacy, Decision-Making Style, Trust, Reliance, Business Simulation

Abstract

This study examines how AI literacy, trust, and decision-making style shape reliance on Generative AI in high-stakes business decisions. Fifty-two participants completed an inventory management task in the MonsoonSIM business simulation, supported by a Generative AI advisor powered by Google Gemini.
We measured AI literacy, trust, decision-making style, and performance expectancy via survey, and reliance, decision quality, and business outcomes from participant responses and game data. AI literacy and performance expectancy significantly predicted reliance and trust, respectively. Avoidant decision-makers showed the strongest tendency to rely on AI, while rational decision-makers remained cautious. Reliance alone did not improve decision quality or business outcomes, but experience in data analysis buffered the negative effect of over-reliance on decision quality. These findings highlight the importance of AI literacy, trust calibration, and analytical competence, rather than reliance itself, in shaping effective human-AI decision-making.

References

Adadi, A., & Berrada, M. (2018). Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access, 6, 52138-52160. https://doi.org/10.1109/ACCESS.2018.2870052

Alvarado-Valencia, J. A., & Barrero, L. H. (2014). Reliance, trust and heuristics in judgmental forecasting. Computers in Human Behavior, 36, 102-113. https://doi.org/10.1016/j.chb.2014.03.047

Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI '21) (pp. 1-16). Association for Computing Machinery. https://doi.org/10.1145/3411764.3445717

Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R.,

Chatila, R., & Herrera, F. (2020). Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, 82-115. https://doi.org/10.1016/j.inffus.2019.12.012

Benda, N. C., Novak, L. L., Reale, C., & Ancker, J. S. (2022). Trust in AI: Why we should be designing for appropriate reliance. Journal of the American Medical Informatics Association, 29(1), 207-212. https://doi.org/10.1093/jamia/ocab238

Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610-623). Association for Computing Machinery. https://doi.org/10.1145/3442188.3445922

Brynjolfsson, E., & McAfee, A. (2014). The second machine age: Work, progress, and prosperity in a time of brilliant technologies. W. W. Norton & Company.

Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), Article 188, 1-21. https://doi.org/10.1145/3449287

Bussone, A., Stumpf, S., & O'Sullivan, D. (2015). The role of explanations on trust and reliance in clinical decision support systems. In 2015 International Conference on Healthcare Informatics (pp. 160-169). IEEE. https://doi.org/10.1109/ICHI.2015.26

Cao, S., & Huang, C.-M. (2022). Understanding user reliance on AI in assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2), Article 471, 1-23. https://doi.org/10.1145/3555572

Chauhan, S., & Jaiswal, M. (2016). Determinants of acceptance of ERP software training in business schools: Empirical investigation using UTAUT model. The International Journal of Management Education, 14(3), 248-262. https://doi.org/10.1016/j.ijme.2016.05.005

Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023). Understanding the role of human intuition on reliance in human-AI decision-making with explanations. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW2), Article 370, 1-32. https://doi.org/10.1145/3610219

Cheng, R., Wang, R., Zimmermann, T., & Ford, D. (2024). “It would work for me too”: How online communities shape software developers’ trust in AI-powered code generation tools. ACM Transactions on Interactive Intelligent Systems, 14(2), 1-39. https://doi.org/10.1145/3651990

Chi, O. H., Chi, C. G., Gursoy, D., & Nunkoo, R. (2023). Customers’ acceptance of artificially intelligent service robots: The influence of trust and culture. International Journal of Information Management, 70, 102623. https://doi.org/10.1016/j.ijinfomgt.2023.102623

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.

Davis, F. D. (1985). A technology acceptance model for empirically testing new end-user information systems: Theory and results [Doctoral dissertation, Massachusetts Institute of Technology]. MIT Open Scholarship. http://hdl.handle.net/1721.1/15192

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319-340. https://doi.org/10.2307/249008

Deldjoo, Y., He, Z., McAuley, J., Korikov, A., Sanner, S., Ramisa, A., Vidal, R., Sathiamoorthy, M., Kasrizadeh, A., & Milano, S. (2024). Recommendation with generative models. arXiv. https://doi.org/10.48550/arXiv.2409.15173

Dellermann, D., Ebel, P., Söllner, M., & Leimeister, J. M. (2019). Hybrid intelligence. Business & Information Systems Engineering, 61(5), 637-643. https://doi.org/10.1007/s12599-019-00595-2

Driver, M. J., Brousseau, K. R., & Hunsaker, P. L. (1998). The dynamic decision maker: Five decision styles for executive and business success. IUniverse.

Eigner, E., & Händler, T. (2024). Determinants of LLM-assisted decision-making. arXiv. https://doi.org/10.48550/arXiv.2402.17385

Glikson, E., & Woolley, A. W. (2020). Human trust in artificial intelligence: Review of empirical research. Academy of Management Annals, 14(2), 627-660. https://doi.org/10.5465/annals.2018.0057

Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121-127. https://doi.org/10.1136/amiajnl-2011-000089

Gursoy, D., Chi, O. H., Lu, L., & Nunkoo, R. (2019). Consumers' acceptance of artificially intelligent (AI) device use in service delivery. International Journal of Information Management, 49, 157-169. https://doi.org/10.1016/j.ijinfomgt.2019.03.008

Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis (7th ed.). Pearson.

Harren, V. A. (1979). A model of career decision making for college students. Journal of Vocational Behavior, 14(2), 119-133. https://doi.org/10.1016/0001-8791(79)90065-4

Hoff, K. A., & Bashir, M. (2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407-434. https://doi.org/10.1177/0018720814547570

Jabotinsky, H. Y., & Sarel, R. (2023). Co-authoring with an AI? Ethical dilemmas and artificial intelligence. Arizona State Law Journal, Forthcoming. https://doi.org/10.2139/ssrn.4303959

Jarrahi, M. H. (2018). Artificial intelligence and the future of work: Human-AI symbiosis in organizational decision making. Business Horizons, 61(4), 577-586. https://doi.org/10.1016/j.bushor.2018.03.007

Keeney, R. L., & Raiffa, H. (1993). Decisions with multiple objectives: Preferences and value tradeoffs. Cambridge University Press. https://doi.org/10.1017/CBO9781139174084

Klingbeil, A., Grützner, C., & Schreck, P. (2024). Trust and reliance on AI: An experimental study on the extent and costs of overreliance on AI. Computers in Human Behavior, 160, 108352. https://doi.org/10.1016/j.chb.2024.108352

Knoth, N., Tolzin, A., Janson, A., & Leimeister, J. M. (2024). AI literacy and its implications for prompt engineering strategies. Computers and Education: Artificial Intelligence, 6, 100225. https://doi.org/10.1016/j.caeai.2024.100225

Körber, M., Baseler, E., & Bengler, K. (2018). Introduction matters: Manipulating trust in automation and reliance in automated driving. Applied Ergonomics, 66, 18-31. https://doi.org/10.1016/j.apergo.2017.07.006

Lankton, N. K., McKnight, D. H., & Tripp, J. (2015). Technology, humanness, and trust: Rethinking trust in technology. Journal of the Association for Information Systems, 16(10), 1. https://doi.org/10.17705/1jais.00411

Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50-80. https://doi.org/10.1518/hfes.46.1.50.30392

Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design considerations. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-16). Association for Computing Machinery. https://doi.org/10.1145/3313831.3376727

Lu, L., Cai, R., & Gursoy, D. (2019). Developing and validating a service robot integration willingness scale. International Journal of Hospitality Management, 80, 36-51. https://doi.org/10.1016/j.ijhm.2019.01.005

Meissner, P., & Keding, C. (2021). The human factor in AI-based decision-making. MIT Sloan Management Review, 63(1), 1-5.

Mohd Khalil, A., Lee, K. L., Kamaruzzaman, Z. A., & Ong, C. A. (2024). Effectiveness of simulation-based learning in Malaysian higher education: A case study of MonsoonSIM. Asian Education and Development Studies, 13(1), 64-77. https://doi.org/10.1108/aeds-09-2023-0125

Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021). Conceptualizing AI literacy: An exploratory review. Computers and Education: Artificial Intelligence, 2, 100041. https://doi.org/10.1016/j.caeai.2021.100041

OECD/European Union/EC-JRC. (2008). Handbook on constructing composite indicators: Methodology and user guide. OECD Publishing. https://doi.org/10.1787/9789264043466-en

Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381-410. https://doi.org/10.1177/0018720810376055

Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230-253. https://doi.org/10.1518/001872097778543886

Phillips, S. D., Pazienza, N. J., & Ferrin, H. H. (1984). Decision-making styles and problem-solving appraisal. Journal of Counseling Psychology, 31(4), 497. https://doi.org/10.1037/0022-0167.31.4.497

Pinski, M., & Benlian, A. (2023). AI literacy-towards measuring human competency in artificial intelligence. In Proceedings of the Annual Hawaii International Conference on System Sciences (pp. 165-174). Hawaii International Conference on System Sciences. https://doi.org/10.24251/hicss.2023.021

Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K., & Succi, M. D. (2023). Assessing the utility of ChatGPT throughout the entire clinical workflow: Development and usability study. Journal of Medical Internet Research, 25, e48659. https://doi.org/10.2196/48659

Ritbumroong, T. (2023). I am therefore, I do: A fit perspective of decision-making styles and business intelligence usage. Journal of Decision Systems, 1-26. https://doi.org/10.1080/12460125.2023.2246251

Rivas, P., & Zhao, L. (2023). Marketing with ChatGPT: Navigating the ethical terrain of GPT-based chatbot technology. AI, 4(2), 375-384. https://doi.org/10.3390/ai4020019

Schemmer, M., Bartos, A., Spitzer, P., Hemmer, P., Kühl, N., Liebschner, J., & Satzger, G. (2023). Towards effective human-AI decision-making: The role of human learning in appropriate reliance on AI advice. arXiv. https://doi.org/10.48550/arXiv.2310.02108

Scott, S. G., & Bruce, R. A. (1995). Decision-making style: The development and assessment of a new measure. Educational and Psychological Measurement, 55(5), 818-831. https://doi.org/10.1177/0013164495055005017

Septiawan, B., Musyaffi, A. M., Quinn, M., & Ratifah, I. (2023). Technology readiness in enterprise resource planning gamification to improve student learning outcomes. International Journal of Evaluation and Research in Education, 12(3), 1375-1382. https://doi.org/10.11591/ijere.v12i3.25226

Shafiai, Z., & Omar, P. U. (2019). 21st century skills development through business simulation games. In 4th International Conference on Research in TVET Studies. Pamulang University (UNPAM).

Shafudin, M. Y., Chin, F. G., & Raihanatul, Z. M. (2018). Factors influencing use of MonsoonSIM business simulation by UTM undergraduate students. International Journal of Learning and Development, 8(2), 61-79. https://doi.org/10.5296/ijld.v8i2.13073

Shin, D. (2021). The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable AI. International Journal of Human-Computer Studies, 146, 102551. https://doi.org/10.1016/j.ijhcs.2020.102551

Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies, 51(5), 991-1006. https://doi.org/10.1006/ijhc.1999.0252

Spetzler, C., Winter, H., & Meyer, J. (2016). Decision quality: Value creation from better business decisions. Wiley.

Tumphasuwan, K. (2023). Using the business digital simulation game of MonsoonSIM in the business process management course for creating knowledge and understanding of university students. In Proceedings of the 2023 7th International Conference on Education and E-Learning (pp. 155-159). Association for Computing Machinery. https://doi.org/10.1145/3637989.3638024

Vaccaro, M., Almaatouq, A., & Malone, T. W. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour, 8(12), 2293-2303. https://doi.org/10.1038/s41562-024-02024-1

Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the technology acceptance model: Four longitudinal field studies. Management Science, 46(2), 186-204. https://doi.org/10.1287/mnsc.46.2.186.11926

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425-478. https://doi.org/10.2307/30036540

Vereschak, O., Bailly, G., & Caramiaux, B. (2021). How to evaluate trust in AI-assisted decision making? A survey of empirical methodologies. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2), Article 327, 1-39. https://doi.org/10.1145/3476068

Weiser, B. (2023, May 27). Here’s what happens when your lawyer uses ChatGPT. The New York Times. https://www.nytimes.com/2023/05/27/nyregion/avianca-airline-lawsuit-chatgpt.html

Williams, M. L., Dennis, A. R., Stam, A., & Aronson, J. E. (2007). The impact of DSS use and information load on errors and decision quality. European Journal of Operational Research, 176(1), 468-481. https://doi.org/10.1016/j.ejor.2005.06.064

Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020). Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Conference on Fairness, Accountability, and Transparency (FAT* '20) (pp. 295-305). Association for Computing Machinery. https://doi.org/10.1145/3351095.3372852