A Hybrid Facial Expression Recognition System Based on Machine Learning and Deep Learning Models

บทคัดย่อ

Pain assessment through facial expressions is an important area of research because many patients cannot clearly communicate their pain levels. This study presents the first systematic comparison of continuous time-series versus tokenized sequence representations for facial action unit (AU)-based pain classification, introducing a novel application of NLP models (BERT) to discretized AU sequences treated as symbolic text. Two datasets were used: the UNBC-McMaster Shoulder Pain Archive (UNBC-SP) with about 48,000 frames, and the Multimodal Intensity Pain (MIntPain) dataset with about 187,900 frames. Action unit intensities were extracted using the Py-Feat library and then normalized, oversampled, and augmented. A range of models was tested, including random forest, support vector machine, recurrent neural networks, and BERT. Key contributions include: (1) demonstrating that continuous time-series models significantly outperform tokenized approaches (91% vs. 82% accuracy); (2) revealing that classical ensemble methods surpass deep learning on tokenized sequences in data-limited scenarios; and (3) establishing that disruptive augmentations harm performance while conservative methods maintain accuracy. The continuous-time series models achieved the best performance, reaching 91% accuracy on MIntPain and 84% on UNBC-SP, while the tokenized models peaked at 82%. The results suggest that preserving temporal details of facial action units provides an advantage for pain detection, especially with larger datasets, though tokenization may retain value in resource-limited settings. The study highlights the need for larger, more diverse datasets and for validation in real clinical settings to improve the reliability of automatic pain recognition.