On the detection of exploitation of vulnerabilities leading to the execution of a malicious code

Yu. V. Kosolapov

Modelirovanie i analiz informacionnyh sistem, Tome 27 (2020) no. 2, pp. 138-151.

Résumé

Software protection from exploitation of possible unknown vulnerabilities can be performed both by searching (for example, using symbolic execution) and subsequent elimination of the vulnerabilities and by using detection and / or intrusion prevention systems. In the latter case, this problem is usually solved by forming a profile of a normal behavior and deviation from normal behavior over a predetermined threshold is regarded as an anomaly or an attack. In this paper, the task is to protect a given software $P$ from exploiting unknown vulnerabilities. For this aim a method is proposed for constructing a profile of the normal execution of the program $P$, in which, in addition to a set of legal chains of system and library functions, it is proposed to take into account the distances between adjacent function calls. At the same time, a profile is formed for each program. It is assumed that taking into account the distances between function calls will reveal shell code execution using system and / or library function calls. An algorithm and a system for detecting abnormal code execution are proposed. The work carried out experiments in the case when $P$ is the FireFox browser. During the experiments the possibility of applying the developed algorithm to identify abnormal behavior when launching publicly available exploits was investigated.

Détail

Keywords: system calls, library calls, software vulnerability.

Bibliographie

[1] A. Khraisat, I. Gondal, P. Vamplew, J. Kamruzzaman, “Survey of intrusion detection systems: techniques, datasets and challenges”, Cybersecurity, 2:1 (2019), 20 | DOI

[2] S. Forrest, S. Hofmeyr, A. Somayaji, “The evolution of system-call monitoring”, Proceedings of 2008 Annual Computer Security Applications Conference (ACSAC), 2008, 418–430 | DOI

[3] S. Gupta, H. Sharma, S. Kaur, “Malware characterization using windows API call sequences”, Journal of Cyber Security and Mobility, 7:4 (2018), 363–378 | DOI

[4] R. Veeramani, N. Rai, “Windows API based malware detection and framework analysis”, International Journal of Scientific Engineering Research, 3:3 (2012), 1–6

[5] A. Singh, R. Arora, H. Pareek, Malware analysis using multiple api sequence mining control flow graph, , 2017 1707.02691

[6] M. L. Bernardi, M. Cimitile, D. Distante, F. Martinelli, F. Mercaldo, “Dynamic malware detection and phylogeny analysis using process mining”, International Journal of Information Security, 18:3 (2019), 257–284 | DOI

[7] L. Viljanen, A survey of application level intrusion detection, Technical report, Series of Publications C, Report C-2004-61, Helsinki, 2004

[8] G. Creech, Developing a high-accuracy cross platform host-based intrusion detection system capable of reliably detecting zero-day attacks, PhD thesis, University of New South Wales, Canberra, Australia, 2014

[9] H. Hu, S. Shinde, S. Adrian, Z. L. Chua, P. Saxena, Z. Liang, “Data-oriented programming: on the expressiveness of non-control data attacks”, 2016 IEEE Symposium on Security and Privacy (SP), 2016, 969–986 | MR

[10] K. K. Ispoglou, B. AlBassam, T. Jaeger, M. Payer, “Block oriented programming: automating data-only attacks”, Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, 2018, 1868–1882 | DOI

[11] Y. V. Kosolapov, “About detection of code reuse attacks”, Modelirovanie i Analiz Informatsionnykh Sistem, 26:2 (2019), 213–228

[12] D. Wagner, P. Soto, “Mimicry attacks on host-based intrusion detection systems”, Proceedings of the 9th ACM Conference on Computer and Communications Security, 2002, 255–264 | DOI

[13] K. Z. Snow, F. Monrose, L. Davi, A. Dmitrienko, C. Liebchen, A. Sadeghi, “Just-in-time code reuse: on the effectiveness of fine-grained address space layout randomization”, 2013 IEEE Symposium on Security and Privacy, 2013, 574–588 | DOI

[14] E. Stalmans, S. El-Sherei, Macro-less code exec in MSWord, (Last access 12.12.2019) https://sensepost.com/blog/2017/macro-less-code-exec-in-msword/

[15] P. D. Borisov, Y. V. Kosolapov, “On the automatic analysis of the practical resistance of obfusting transformations”, Modelirovanie i Analiz Informatsionnykh Sistem, 26:3 (2019), 317–331 | MR

[16] API Monito, (Last access 28.11.2019) http://www.rohitab.com/apimonitor

[17] ListDLLs, (Last access 28.11.2019) https://docs.microsoft.com/en-us/sysinternals/downloads/listdlls

[18] M. Vervier, M. Orru, B. J. Wever, E. Sesterhenn, Browser security whitepaper, (Last access 05.12.2019) https://browser-security.x41-dsec.de/X41-Browser-Security-White-Paper.pdf

[19] R. Gawlik, T. Holz, “Sok: make JIT-spray great again”, WOOT'18 Proceedings of the 12th USENIX Conference on Offensive Technologies, 2018, 1–14

[20] Offensive security, (Last access 05.12.2019) https://github.com/offensive-security/exploitdb/blob/master/exploits/windows/remote/42484.html

[21] 0vercl0k, CVE-2019-9810, (Last access 05.12.2019) https://github.com/0vercl0k/CVE-2019-9810

[22] Exploit database, (Last access 05.12.2019) https://www.exploit-db.com/

[23] CVE-2017-5375_ASM.JS_JIT-Spray, (Last access 30.12.2019) https://github.com/rh0dev/expdev/tree/master