CTPG-Net: A Cross-Modal Image-Text Matching Network for Traffic Police Gesture Understanding

Abstract

In complex traffic environments, autonomous driving systems not only need to recognize traditional traffic signs and traffic lights, but also need to understand the semantics of traffic police gestures accurately. Accurate interpretation of traffic police gestures is crucial for ensuring road safety and improving traffic efficiency. However, most existing research treats traffic police gestures as a single classification task, primarily focusing on gesture category recognition and neglecting the fine-grained semantics behind these gestures, thereby failing to meet the semantic understanding requirements of intelligent transportation systems. To address this issue, we propose a cross-modal image-text semantic-matching framework for understanding traffic police gestures. We construct a cross-modal dataset of traffic police gestures (CTPG-Dataset), equipping each gesture image with a fine-grained textual semantic description. We also design a two-stream cross-modal matching network (CTPG-Net), which combines BERT and ResNet50, introduces a Top-K Hard Negative InfoNCE (THN-InfoNCE) loss function, and improves cross-modal feature representation and semantic discrimination by mining the most challenging hard-negative samples and incorporating a temperature coefficient optimization strategy. We conducted extensive comparative and ablation experiments, and the results show that the combination of ResNet50 and BERT achieves the best performance when THN-InfoNCE optimization is introduced. Our method achieves an mAP of 79.56% and shows significant improvements in metrics such as Recall@1 and MRR. Inference experiments further validate that the model can accurately perform image-to-text and text-to-image retrieval, demonstrating strong cross-modal semantic understanding and robust retrieval.