Yildiz, A., Teo, S. G., Lou, Y., Feng, Y., Wang, C., & Divakaran, D. M. (2025). Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 30848–30865. https://doi.org/10.18653/v1/2025.acl-long.1490
Abstract:
Large Language Models (LLMs) have shown promise in software vulnerability detection, particularly on function-level benchmarks like Devign and BigVul. However, real-world detection requires interprocedural analysis, as vulnerabilities often emerge through multi-hop function calls rather than isolated functions. While repository-level benchmarks like ReposVul and VulEval introduce interprocedural context, they remain computationally expensive, lack pairwise evaluation of vulnerability fixes, and explore limited context retrieval, limiting their practicality.
We introduce JITVUL, a JIT vulnerability detection benchmark linking each function to its vulnerability-introducing and fixing commits. Built from 879 CVEs spanning 91vulnerability types, JITVUL enables
comprehensive evaluation of detection capabilities. Our results show that ReAct Agents, leverag-
ing thought-action-observation and interproce-dural context, perform better than LLMs in
distinguishing vulnerable from benign code. While prompting strategies like Chain-of-
Thought help LLMs, ReAct Agents require further refinement. Both methods show in-
consistencies, either misidentifying vulnerabilities or over-analyzing security guards, in-
dicating significant room for improvement.
License type:
Publisher Copyright
Funding Info:
There was no specific funding for the research done