Zhang, X., Dong, Y., Wu, Y., Huang, J., Jia, C., Fernando, B., Shou, M. Z., Zhang, L., & Liu, J. (2025). PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning. In (Editor), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://doi.org/10.18653/v1/2025.acl-long.811
Abstract:
Large language models demonstrate remarkable capabilities across various domains, especially mathematics and logic reasoning. However, current evaluations overlook physics-based reasoning - a complex task requiring physics theorems and constraints. We present PhysReason, a 1,200-problem benchmark comprising knowledge-based (25%) and reasoning-based (75%) problems, where the latter are divided into three difficulty levels (easy, medium, hard). Notably, problems require an average of 8.1 solution steps, with hard requiring 15.6, reflecting the complexity of physics-based reasoning. We propose the Physics Solution Auto Scoring Framework, incorporating efficient answer-level and comprehensive step-level evaluations. Top-performing models like Deepseek-R1, Gemini-2.0-Flash-Thinking, and o3-mini-high achieve less than 60% on answer-level evaluation, with performance dropping from knowledge questions (75.11%) to hard problems (31.95%). Through step-level evaluation, we identified four key bottlenecks: Physics Theorem Application, Physics Process Understanding, Calculation, and Physics Condition Analysis. These findings position PhysReason as a novel and comprehensive benchmark for evaluating physics-based reasoning capabilities in large language models.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the National Natural Science Foundation of China - NA
Grant Reference no. : 62137002; 62293550; 62293553; 62293554; 62450005; 62437002; 62477036; 62477037; 62176209; 62192781; 62306229
This research / project is supported by the National Key Research and Development Program of China - NA
Grant Reference no. : 2022YFC3303600
This research / project is supported by the Shaanxi Provincial Social Science Foundation - NA
Grant Reference no. : 2024P041
This research / project is supported by the Natural Science Basic Research Program of Shaanxi - NA
Grant Reference no. : 2023-JC-YB-593
This research / project is supported by the National Research Foundation, Singapore - NRF Fellowship
Grant Reference no. : NRF-NRFF14-2022-0001