On the automatic analysis of the practical resistance of obfusting transformations

P. D. Borisov ; Yu. V. Kosolapov

Modelirovanie i analiz informacionnyh sistem, Tome 26 (2019) no. 3, pp. 317-331.

Résumé

A method is developed for assessing the practical persistence of obfuscating transformations of programs based on the calculation of the similarity index for the original, obfuscated and deobfuscated programs. Candidates are proposed for similarity indices, which are based on such program characteristics as the control flow graph, symbolic execution time and degree of coverage for symbolic execution. The control flow graph is considered as the basis for building other candidates for program similarity indicators. On its basis, a new candidate is proposed for the similarity index, which, when calculated, finds the Hamming distance between the adjacency matrices of control flow graphs of compared programs. A scheme for estimating (analyzing) the persistence of obfuscating transformations is constructed, according to which for the original, obfuscated and deobfuscated programs, the characteristics of these programs are calculated and compared in accordance with the chosen comparison model. The developed scheme, in particular, is suitable for comparing programs based on similarity indices. This paper develops and implements one of the key units of the constructed scheme — a block for obtaining program characteristics compiled for the x86/x86_64 architecture. The developed unit allow to find the control flow graph, the time for symbolic execution and the degree of coverage for symbolic execution. Some results of work of the constructed block are given.

Détail

Keywords: code obfuscation, resistance, symbolic execution.

Bibliographie

[1] Siegmund J., “Program Comprehension: Past, Present, and Future”, IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), v. 5, 2016, 13–20

[2] Avidan E., Feitelson D. G., “From Obfuscation to Comprehension”, Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension, 2015, 178–181 | DOI

[3] Pozdeev A. G., Krivopalov V. N., Romashkin E. V., Radchenko E. D., “Matematicheskie i programmnye sredstva obfuskacii programm”, PDM, 1 (2009), 52–53 (in Russian)

[4] Chernov A. V., Analiz zaputyvayushchih preobrazovanij programm, 2002 (in Russian)

[5] Kuzurin N., Shokurov A., Varnovsky N., Zakharov V., “On the Concept of Software Obfuscation in Computer Security”, International Conference on Information Security, Lecture Notes in Computer Science, 4779, Springer, Berlin–Heidelberg, 2007, 281–298 | DOI | Zbl

[6] Diffie W., Hellman M., “New directions in cryptography”, IEEE Transactions on Information Theory, 22:6 (1976), 644–654 | DOI | MR | Zbl

[7] Collberg C. S., Thomborson C., “Watermarking, Tamper-Proofing, and Obfuscation Tools for Software Protection”, IEEE transactions on software engineering, 28:8 (2002), 735–746 | DOI

[8] Lee B., Kim Y., Kim J., “binOb+: a Framework for Potent and Stealthy Binary Obfuscation”, Proceedings of the 5th ACM Symposium on Information, Computer and Communications Security, ASIACCS 2010, 2010, 271–281

[9] Borello J. M., Me L., “Code Obfuscation Techniques for Metamorphic Viruses”, Journal in Computer Virology, 4:3 (2008), 211–220 | DOI

[10] Moser A., Kruegel C., Kirda E., “Limits of Static Analysis for Malware Detection”, Proceedings of Twenty-Third Annual Computer Security Applications Conference, ACSAC 2007, 2007, 421–430 | DOI

[11] Baiardi F., Sgandurra D., “An obfuscation-based approach against injection attacks”, Proceedings of the Sixth International Conference on Availability, Reliability and Security, ARES, 2011, 51–58 | DOI

[12] Nurmukhametov A. R., “Primenenie diversificiruyushchih i obfusciruyushchih preobrazovanij dlya izmeneniya signatury programmnogo koda”, Trudy ISP RAN, 28:5 (2016), 93–104 (in Russian)

[13] Kosolapov Y. V., “About Detection of Code Reuse Attacks”, Modeling and Analysis of Information Systems, 26:2 (2019), 213–228 (in Russian)

[14] Collberg C., Thomborson C., Low D., A taxonomy of obfuscating transformations, Technical Report 148, The University of Auckland, New Zealand, 1997

[15] Walenstein A., El-Ramly M., Cordy J. R., Evans W. S., Mahdavi K., Pizka M., Ramalingam G., von Gudenberg J. W., “Similarity in Programs”, Duplication, Redundancy, and Similarity in Software, 2007, 1–8

[16] Chipounov V., Kuznetsov V., Candea G., “The S2E Platform: Design, Implementation, and Applications”, ACM Transactions on Computer Systems, 30:1 (2012), 1–49 | DOI

[17] Saudel F., Salwan J., “Triton: A Dynamic Symbolic Execution Framework”, Symposium Sur La Security Des Technologies de L'information et Des Communications, SSTIC, 2015, 31–54

[18] Wang Z, Ming J., Jia C., Gao D., “Linear Obfuscation to Combat Symbolic Execution”, Proceedings of Computer Security – ESORICS 2011, Lecture Notes in Computer Science, 6879, 2011, 210–226 | DOI | MR

[19] Brumley D., Hartwig C., Liang Z., Newsome J., Song D., Yin H., “Automatically Identifying Trigger-based Behavior in Malware”, Botnet Detection, Advances in Information Security, 36, 2008, 65–88

[20] King J. C., “Symbolic execution and program testing”, Communications of the ACM, 19:7 (1976), 385–394 | DOI | MR | Zbl

[21] Cadar C., Dunbar D., Engler D. R., “KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs”, 8th USENIX Symposium on Operating Systems Design and Implementation, v. 8, 2008, 209–224

[22] Shoshitaishvili Y.et al., “SoK: (State of) The Art of War: Offensive Techniques in Binary Analysis”, IEEE Symposium on Security and Privacy, 2016, 138–157

[23] Sharif M. I., Lanzi A., Giffin J. T., Lee W., “Impeding Malware Analysis Using Conditional Code Obfuscation”, Proceedings of NDSS, 2008, 1–13

[24] Udupa S. K., Debray S. K., Madou M., “Deobfuscation: Reverse Engineering Obfuscated Code”, Proceedings of the 12th Working Conference on Reverse Engineering, WCRE'05, 2005, 44–53

[25] Nagarajan V., Gupta R., Zhang X., Madou M., De Sutter B., “Matching Control Flow of Program Versions”, IEEE International Conference on Software Maintenance, 2007, 84–93

[26] Bonfante G., Kaczmarek M., Marion J. Y., “Control Flow Graphs as Malware Signatures”, International Workshop on the Theory of Computer Viruses, 2007, 1–6

[27] Park Y., Reeves D., Mulukutla V., Sundaravel B., “Fast Malware Classification by Automated Behavioral Graph Matching”, Proceedings of the Sixth Annual Workshop on Cyber Security and Information Intelligence Research, 2010, 1–4

[28] Kinable J., Kostakis O., “Malware classification based on call graph clustering”, Journal in Computer Virology, 7:4 (2011), 233–245 | DOI

[29] Lim H.I., “Comparing Control Flow Graphs of Binary Programs through Match Propagation”, IEEE 38th Annual Computer Software and Applications Conference, 2014, 598–599

[30] Dullien T., Rolles R., “Graph-based comparison of executable objects”, SSTIC 2005, 1–8

[31] Chan P.P.F., Collberg C., “A Method to Evaluate CFG Comparison Algorithms”, 14th International Conference on Quality Software, 2014, 95–104

[32] Axenovich M., Kezdy A., Martin R., “On the editing distance of graphs”, J. Graph Theory, 58:2 (2008), 123–138 | DOI | MR | Zbl

[33] Borisov P. D., Kosolapov Y. V., “O vybore harakteristik dlja ocenki stojkosti obfuscirujushhih preobrazovanij”, Sovremennye informacionnye tekhnologii: tendencii i perspektivy razvitiya, Trudy XXV nauchnoj konferencii SITO-2019, 2019, 42–44 (in Russian)

[34] Lehman M. M., Belady L. A., PProgram Evolution. Processes of Software Change, Academic press, 1985, 539 pp.

[35] Schnappinger M., Osman M. H., Pretschner A., Pizka M., Fietzke A., “Software Quality Assessment in Practice: a Hypothesis-Driven Framework”, Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 2018, 1–6

[36] Borisov P. D., Kosolapov Y. V., “Model' eksperimental'nogo analiza stojkosti algoritmov obfuskacii”, Sovremennye informacionnye tekhnologii: tendencii i perspektivy razvitiya, Trudy XXV nauchnoj konferencii SITO-2018, 2018, 37–39 (in Russian)

[37] IDA Pro, https://www.hex-rays.com/products/ida/

[38] The LLVM Compiler Infrastructure, https://llvm.org/

[39] McSema, https://github.com/trailofbits/mcsema

[40] Junod P., Rinaldini J., Wehrli J., Michieliny J., “Obfuscator-LLVM – Software Protection for the Masses”, Conference: 2015 IEEE/ACM 1st International Workshop on Software Protection (SPRO), 2015, 3–9