On the study of one way to detect anomalous program execution

Yu. V. Kosolapov ; T. A. Pavlova

Modelirovanie i analiz informacionnyh sistem, Tome 31 (2024) no. 2, pp. 152-163.

Résumé

Developing more accurate and adaptive methods for detecting malicious code is a critical challenge in the context of constantly evolving cybersecurity threats. This requires constant attention to new vulnerabilities and attack methods, as well as the search for innovative approaches to detecting and preventing cyber threats. The paper examines an algorithm for detecting the execution of malicious code in the process of a protected program. This algorithm is based on a previously proposed approach, when the legitimate execution of a protected program is described by a profile of differences in the return addresses of called functions, also called a distance profile. A concept has been introduced called positional distance, which is determined by the difference between the call numbers in the program trace. The main change was the ability to add to the profile the distances between the return addresses of not only neighboring functions, but also several previous ones with a given positional distance. In addition to modifying the detection algorithm, the work developed a tool for automating the construction of a distance profile and experimentally studied the dependence of the probability of false detection of an atypical distance on the training duration for four well-known browsers. Experiments confirm that with a slight increase in verification time, the number of atypical distances detected by the proposed algorithm can be significantly less than the number of atypical distances detected by the basic algorithm. However, it should be noted that the effect of the transition from the basic algorithm to the proposed one, as the results showed, depends on the characteristics of the specific program being protected. The study highlights the importance of continually improving malware detection techniques to adapt them to changing threats and software operating conditions. As a result, this will ensure more reliable protection of information and systems from cyber attacks and other cyber threats.

Détail

Mots-clés : exploits
Keywords: program protection, abnormal program execution.

Bibliographie

[1] Lee, Kyungroul and Lee, Jaehyuk and Yim, Kangbin, “Classification and analysis of malicious code detection techniques based on the APT attack”, Applied Sciences, 13:5 (2023), 2894 | DOI

[2] Hofmeyr, A. and Forrest, S. and Somayaji, A., “Intrusion detection using sequences of system calls”, Journal of computer security, 6:3 (1998), 151–180 | DOI

[3] Wagner, D. and Soto, P., “Mimicry attacks on host-based intrusion detection systems”, Proceedings of the 9th ACM conference on Computer and communications security, 2002, 255–264 | DOI

[4] Kosolapov, Yu., “On one method for detecting exploitation of vulnerabilities and its parameters”, Systems and Means of Informatics, 31:4 (2021), 48–60 (in Russian)

[5] Kosolapov, Yu., “On the Detection of Exploitation of Vulnerabilities That Leads to the Execution of a Malicious Code”, Automatic Control and Computer Sciences, 55 (2021), 827–837 | DOI

[6] Rohitab Batra, API monitor, 2013 (2024-04-21) http://www.rohitab.com/apimonitor

[7] Kechahmadze, A. and Kosolapov, Yu., “Method for detecting exploits based on the profile of differences between function call addresses”, Informatika i sistemy upravleniya, 73:3 (2022), 106–116 (in Russian) | DOI

[8] Exploit Protection Reference, 2023 (2024-04-21) https://learn.microsoft.com/en-us/microsoft-365/security/defender-endpoint/exploit-protection-reference?view=o365-worldwide

[9] Sweigart, Al, PyAutoGUI documentation, 2021 (2024-04-21) https://readthedocs.org/projects/pyautogui/downloads/pdf/latest/

[10] Ding, Yu and Wei, Tao and Xue, Hui and Zhang, Yulong and Zhang, Chao and Han, Xinhui, “Accurate and efficient exploit capture and classification”, Science China. Information Sciences, 60 (2017), 052110:1–052110:17 | DOI