Geodesic
Application of x86 extensions for code protection
Prikladnaya Diskretnaya Matematika. Supplement, no. 14 (2021), pp. 138-140.

Voir la notice de l'article provenant de la source Math-Net.Ru

A new approach is proposed to protect the program code against reverse engineering tools, such as decompilers and symbolic execution tools. The approach is based on the usage of uncommon x86 processor instructions that could be implemented incorrectly in the aforementioned tools. Existing approaches to this problem are also considered, and the relative performance advantage of the proposed approach is noted. A method for numeric constants obfuscation, following this approach, is developed with the usage of AES-NI extension for the x86 architecture and its AESENC instruction in particular. This method is implemented for Clang compiler with the help of LLVM Intermediate Representation and tested against reverse engineering tools, such as IDA and Ghidra decompilers and angr symbolic execution tool.
Mots-clés : code protection
Keywords: reverse engineering, decompiler, symbolic execution, x86 processor architecture. 
@article{PDMA_2021_14_a30,
     author = {R. K. Lebedev and I. A. Koryakin},
     title = {Application of x86 extensions for code protection},
     journal = {Prikladnaya Diskretnaya Matematika. Supplement},
     pages = {138--140},
     publisher = {mathdoc},
     number = {14},
     year = {2021},
     language = {ru},
     url = {http://geodesic.mathdoc.fr/item/PDMA_2021_14_a30/}
}
TY  - JOUR
AU  - R. K. Lebedev
AU  - I. A. Koryakin
TI  - Application of x86 extensions for code protection
JO  - Prikladnaya Diskretnaya Matematika. Supplement
PY  - 2021
SP  - 138
EP  - 140
IS  - 14
PB  - mathdoc
UR  - http://geodesic.mathdoc.fr/item/PDMA_2021_14_a30/
LA  - ru
ID  - PDMA_2021_14_a30
ER  - 
%0 Journal Article
%A R. K. Lebedev
%A I. A. Koryakin
%T Application of x86 extensions for code protection
%J Prikladnaya Diskretnaya Matematika. Supplement
%D 2021
%P 138-140
%N 14
%I mathdoc
%U http://geodesic.mathdoc.fr/item/PDMA_2021_14_a30/
%G ru
%F PDMA_2021_14_a30
R. K. Lebedev; I. A. Koryakin. Application of x86 extensions for code protection. Prikladnaya Diskretnaya Matematika. Supplement, no. 14 (2021), pp. 138-140. http://geodesic.mathdoc.fr/item/PDMA_2021_14_a30/

[1] Junod P., Rinaldini J., Wehrli J., Michielin J., “Obfuscator-LLVM — software protection for the masses”, 2015 IEEE/ACM 1st Intern. Workshop Software Protection, 2015, 3–9

[2] Wang Z., Ming J., Jia C., Gao D., “Linear obfuscation to combat symbolic execution”, Proc. European Symp. Research Computer Security, 2011, 210–226

[3] Seto T., Monden A., Yucel Z., Kanzaki Y., “On preventing symbolic execution attacks by low cost obfuscation”, 20th IEEE/ACIS Intern. Conf. Software Eng., Artif. Intelligence, Networking and Parallel/Distributed Comput. (SNPD), 2019, 495–500

[4] Lebedev R. K., “Avtomaticheskaya generatsiya khesh-funktsii dlya obfuskatsii programmnogo koda”, Prikladnaya diskretnaya matematika, 2020, no. 50, 102–117 | Zbl

[5] Intel XED, , 2019 https://intelxed.github.io/

[6] Intel\circledR Advanced Encryption Standard Instructions (AES-NI), , 2012 https://software.intel.com/content/www/us/en/develop/articles/intel-advanced-encryption-standard-instructions-aes-ni.html

[7] Lattner C., Adve V., “LLVM: A compilation framework for lifelong program analysis transformation”, Intern. Symp. Code Generation and Optimization, 2004, 75–86

[8] IDA Pro, , 2021 https://www.hex-rays.com/ida-pro/

[9] Ghidra Software Reverse Engineering Framework, , 2021 https://github.com/NationalSecurityAgency/ghidra

[10] Shoshitaishvili Y., Wang R., Salls C., et al., “SOK: (State of) The art of war: Offensive techniques in binary analysis”, IEEE Symp. Security Privacy, 2016, 138–157