IoT-Based Sign Language Recognition System

Abstract

This research aims to develop a real-time English alphabet sign language translation system based on a hybrid computing approach integrating a web client and an IoT device. The system employs MediaPipe to extract 42 hand landmark coordinates instead of raw images in order to reduce data complexity and improve processing efficiency. The extracted coordinates are classified using a lightweight Multilayer Perceptron (MLP) model deployed on an ESP32 microcontroller. To address the differences between left and right hands, the Canonical Form concept is applied by transforming right-hand inputs into a unified representation. Experimental results indicate that the proposed system can accurately recognize English alphabet signs from A to Z, excluding J and Z, achieving an average accuracy of 91.67% while maintaining stable performance under varying environmental conditions. The results demonstrate that combining structural hand features with a hybrid computing architecture effectively enhances recognition accuracy and is well suited for IoT-based sign language translation systems with limited computational resources.