A Hybrid Framework Combining Stylometric, Semantic, and SBERT Features for AI-Generated Text Detection

Abstract

The rapid advancement of Large Language Models (LLMs) has greatly improved the ability of artificial intelligence systems to generate human-like text, making AI-generated text detection more challenging. This study introduces a hybrid framework that merges stylometric and semantic features with Sentence-BERT (SBERT) embeddings to enhance classification performance and robustness. For evaluation, experiments used the AI vs Human Text Dataset. Documents under 50 words were excluded to ensure data quality. The final dataset was balanced, with 100,000 documents evenly split between AI-generated and human-written texts. Stylometric features captured writing style, while semantic features measured sentiment, readability, information density, and lexical. SBERT embeddings captured deep contextual semantics. To improve computational efficiency, Principal Component Analysis (PCA) at 95% variance preservation reduced dimensionality. Four machine learning algorithms—Logistic Regression, Support Vector Machine (SVM), XGBoost, and LightGBM—were tested with 10-fold cross-validation. The hybrid framework combined with XGBoost performed best, achieving 98.63% accuracy, a macro F1-score of 0.9863, and an ROC-AUC of 0.9987. The Wilcoxon Signed-Rank Test confirmed that the approach significantly outperformed stylometric, semantic, and SBERT-only representations (p-value = 0.0020). Findings show that combining stylometric, semantic, and contextual embedding features captures both surface-level patterns and deep semantics, providing a reliable solution for detecting AI-generated text.