Malware Analysis and Detection Using Deep Learning via Artificial Neural Networks

Abstract

This study aimed to (1) develop a deep learning model using a Fully Connected Neural Network (FCNN) for malware detection and analysis, (2) investigate the behaviors and attack patterns of various types of malware such as brute-force, phishing, ransomware, and DDoS based on data from Remote Desktop Protocol (RDP) and AnyDesk software, and (3) evaluate the performance and accuracy of the proposed model in detecting anomalous behaviors across diverse environments. The tools used in this research included AnyDesk, RDP protocol, TensorFlow, Keras, and Wireshark. Statistical metrics used to assess model performance were Accuracy, Precision, Recall, and F1-score.

The results showed that (1) the developed FCNN model effectively detected anomalous behaviors from data captured via RDP and AnyDesk, achieving a maximum accuracy of 91.49%; (2) performance evaluation on the test dataset indicated that the model achieved a Precision of 95.45%, Recall of 87.50%, and F1-score of 91.30%, demonstrating a balanced trade-off between detection accuracy and false-positive rate; and (3) the model was capable of accurately analyzing and identifying distinct malware attack behaviors such as brute-force attacks, phishing, ransomware, and DDoS thus significantly reducing cybersecurity risks in real-world environments.