Geodesic
Using x86 mode switching for program code protection
Prikladnaâ diskretnaâ matematika, no. 3 (2023), pp. 104-120.

Voir la notice de l'article provenant de la source Math-Net.Ru

A novel program code obfuscation approach involving the x86 mode switching is proposed in the paper. The details and existing applications of x86 mode switching are reviewed, as well as the possible consequences of using this switching to the reverse engineering tools. Based on this approach, a few specific methods are proposed and evaluated against the most popular reverse engineering tools of various purposes, including disassemblers, decompilers, binary instrumentation and symbolic execution tools. A method of seamless integration of these machine code level obfuscations to the C, C++ and possibly other compilers is also proposed.
Keywords: reverse engineering, obfuscation, x86 mode switching, disassembly, decompilation, symbolic execution.
Mots-clés : code protection 
@article{PDM_2023_3_a5,
     author = {R. K. Lebedev},
     title = {Using x86 mode switching for program code protection},
     journal = {Prikladna\^a diskretna\^a matematika},
     pages = {104--120},
     publisher = {mathdoc},
     number = {3},
     year = {2023},
     language = {en},
     url = {http://geodesic.mathdoc.fr/item/PDM_2023_3_a5/}
}
TY  - JOUR
AU  - R. K. Lebedev
TI  - Using x86 mode switching for program code protection
JO  - Prikladnaâ diskretnaâ matematika
PY  - 2023
SP  - 104
EP  - 120
IS  - 3
PB  - mathdoc
UR  - http://geodesic.mathdoc.fr/item/PDM_2023_3_a5/
LA  - en
ID  - PDM_2023_3_a5
ER  - 
%0 Journal Article
%A R. K. Lebedev
%T Using x86 mode switching for program code protection
%J Prikladnaâ diskretnaâ matematika
%D 2023
%P 104-120
%N 3
%I mathdoc
%U http://geodesic.mathdoc.fr/item/PDM_2023_3_a5/
%G en
%F PDM_2023_3_a5
R. K. Lebedev. Using x86 mode switching for program code protection. Prikladnaâ diskretnaâ matematika, no. 3 (2023), pp. 104-120. http://geodesic.mathdoc.fr/item/PDM_2023_3_a5/

[1] Barak B., Goldreich O., Impagliazzo R., et al., “On the (im)possibility of obfuscating programs”, LNCS, 2139, 2001, 1–18 | Zbl

[2] Cpp2IL: Work-in-progress tool to reverse unity's IL2CPP toolchain, , 2023 https://github.com/SamboyCoding/Cpp2IL

[3] Wang C., Davidson J., Hill J., and Knight J., “Protection of software-based survivability mechanisms”, Proc. Intern. Conf. Dependable Syst. Networks (Goteborg, 2001), 193–202

[4] Collberg C., Thomborson C., and Low D., “Manufacturing cheap, resilient, and stealthy opaque constructs.”, Proc. 25th ACM SIGPLAN-SIGACT Symp. POPL'98 (San Diego, California, USA, 1998), 184–196

[5] Collberg C., Thomborson C., and Low D., “Breaking abstractions and unstructured data structures”, Proc. Intern. Conf. Computer Languages (Chicago, IL, USA, 1998), 28–38

[6] Warren H. S., Hacker's Delight, Second Ed., Addison-Wesley, 2012, 512 pp.

[7] Junod P., Rinaldini J., Wehrli J., and Michielin J., “Obfuscator-LLVM — software protection for the masses”, IEEE/ACM 1st Intern. Workshop Software Protection (Florence, Italy, 2015), 3–9

[8] the tigress c obfuscator, , 2023 https://tigress.wtf

[9] Ugarte-Pedrero X., Balzarotti D., Santos I., and Bringas P. G., “SoK: deep packer inspection: A longitudinal study of the complexity of run-time packers”, EEE Symp. Security and Privacy (San Jose, CA, USA, 2015), 659–673

[10] Jamthagen C., Lantz P., and Hell M., “A new instruction overlapping technique for anti-disassembly and obfuscation of x86 binaries”, Workshop Anti-malware Testing Research (Montreal, QC, Canada, 2013), 1–9

[11] Cohen F. B., “Operating system protection through program evolution”, Computers and Security, 12:6 (1993), 565–584 | DOI

[12] Lebedev R. K. and Koryakin I. A., “Application of x86 extensions for code protection”, Prikladnaya diskretnaya matematika. Prilozhenie, 2021, no. 14, 138–140 (in Russian) | DOI

[13] WoW64 internals, , 2018 https://wbenny.github.io/2018/11/04/wow64-internals.html | Zbl

[14] Understanding Win 7 x64 GDT/LDT, , 2013 https://community.osr.com/discussion/246643

[15] Linux Kernel (GitHub), , 2023 https://github.com/torvalds/linux/blob/master/arch/x86/kernel/cpu/common.c

[16] Intel 64 and IA-32 Architectures Software Developer's Manual Combined Volumes: 1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D, and 4, 2022, 5060 pp.

[17] Allow Different Instruction Sets for Different Memory Sections (Ghidra, GitHub), , 2023 https://github.com/NationalSecurityAgency/ghidra/issues/510

[18] Assembly language is too high level, DEF CON 25, 2017 https://media.defcon.org/DEFCON25/DEFCON25presentations/DEFCON25-XlogicX-Assembly-Language-Is-Too-High-Level.pdf

[19] Collberg C., Thomborson C., and Low D., A Taxonomy of Obfuscating Transformations, Technical Report, No 148, Department of Computer Science, The University of Auckland, 1997 https://researchspace.auckland.ac.nz/bitstream/handle/2292/3491/TR148.pdf

[20] Hex Rays — State-of-the-art binary code analysis solutions, , 2023 https://hex-rays.com/ida-pro

[21] Ghidra Software Reverse Engineering Framework (GitHub), , 2023 https://github.com/NationalSecurityAgency/ghidra

[22] Nethercote N., and Seward J., “Valgrind: a framework for heavyweight dynamic binary instrumentation”, SIGPLAN Not., 42:6 (2007), 89–100 | DOI

[23] Shoshitaishvili Y., Wang R., Salls C., et al., “SOK: (State of) The art of war: Offensive techniques in binary analysis”, IEEE Symp. Security Privacy (SP) (San Jose, CA, USA, 2016), 138–157

[24] Add generalized aam and aad instructions for x86, angr/pyvex (GitHub), , 2022 https://github.com/angr/pyvex/commit/46049a14985a8d78c6679d75f103540b94c22bc5