Ensuring trustworthy code: leveraging a static analyzer to identify and mitigate defects in generated code

D. Shaikhelislamov ; M. Drobyshevskiy ; A. Belevancev

Zapiski Nauchnykh Seminarov POMI, Investigations on applied mathematics and informatics. Part IV, Tome 540 (2024), pp. 233-251

Résumé

The rise of large language models (LLMs) has greatly advanced code generation capabilities. A recent StackOverflow survey found that 70% of developers are using or planning to use AI coding tools this year. However, most current methods focus on supervised fine-tuning objectives derived from text generation, often overlooking the distinct sequence-level properties of code, such as compilability, and syntactic and functional correctness. To address this gap, we introduce a novel approach that combines pre-trained LLMs with software analysis tools commonly used to detect vulnerabilities and validate code. Our method leverages detailed feedback from code compilation and analysis, incorporating this specialized knowledge into the prompt chaining process. We present CodePatchLLM, an extension of LLMs that uses Svace feedback for improved code generation. CodePatchLLM is a model-agnostic framework that supports multiple programming languages. Extensive experiments on the LeetCode dataset show that our approach outperforms the baseline CodeLlama model, achieving significant improvements in compilation success rates and functional correctness across Java, Python, and Kotlin. The CodePatchLLM framework is available at https://github.com/dsshay/CodePatchLLM.