EFFICACY ASSESSMENT OF A RAG-BASED CHATBOT FOR EDUCATIONAL COMPUTER PROGRAMMING

บทคัดย่อ

The purposes of the research were to 1) develop an educational Python chatbot utilizing the open-source Large Language Model, Gemma2 9B, combined with Retrieval-Augmented Generation (RAG) to provide personalized, non-intimidating support for novice programmers facing complex abstract concepts, 2) implement a secure, locally-hosted AI system using Ollama and Docker, managed via OpenWebUI, to ensure a context-aware knowledge base specifically tailored for Python programming instruction, and 3) evaluate the performance and efficacy of the chatbot through a multi-dimensional assessment focusing on semantic consistency, response accuracy via BERT Score, and linguistic fluency via BLEU Score.

The research findings showed that the RAG-based chatbot significantly outperformed the baseline (No-RAG) model across all evaluation dimensions. It achieved a high Consistency score of 0.96 compared to the baseline's 0.82, indicating highly stable and predictable responses. Furthermore, the system demonstrated robust semantic understanding with an impressive BERT Score (Precision: 0.90, Recall: 0.88, F1-score: 0.89), successfully capturing the relevance of educational content compared to the baseline's F1-score of 0.76. However, while the BLEU Score of 0.29 was an improvement over the baseline's 0.21, it highlighted limitations in generating nuanced, human-like linguistic responses. Ultimately, integrating external knowledge via RAG provides a reliable, accurate, and powerful solution for decentralized educational tools, though further optimization in prompt engineering is recommended to enhance natural language flow. The evaluation was conducted using a benchmark dataset consisting of 50 Python programming questions; therefore, the findings should be interpreted within the scope of this dataset.