Development of a Context-Aware Customer Service Chatbot with Automated Tool Selection Using Large Language Models and LangChain

Abstract

Customer service organizations often face challenges in managing fragmented information scattered across internal systems and various types of documents. These issues are further compounded by the loss of procedural knowledge resulting from high employee turnover. This study presents the design and development of an automated chatbot system that integrates a large language model (LLM) with two categories of external tools: Retrieval-Augmented Generation (RAG) for extracting information from documents, and APIs for accessing external systems. The system is built on the LangChain framework, which enables agent orchestration and modular integration of tools. The RAG-based tools are designed to implement a hybrid search approach, combining keyword-based and semantic retrieval, with result ranking performed using Reciprocal Rank Fusion to identify the most relevant information. The chatbot is capable of automatically selecting the appropriate tools based on the context of each user query. Evaluation results show that the system achieved a tool call accuracy of 0.88 and an answer correctness rate of 0.80. Additionally, user and staff satisfaction rates were reported at 85.77% and 85.04%, respectively, highlighting the system’s effectiveness in addressing data retrieval complexity and facilitating knowledge integration in service environments.