Fang, F., Yang, M., Wu, M., Yang, Y., Xu, Q., Lim, J.-H., Yang, X., & Zhu, H. (2026). Toward Accurate Procedure Planning in Instructional Videos: Visual State Generation Helps Task-Selective Diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(4), 4033–4050. https://doi.org/10.1109/tpami.2025.3641798
Abstract:
Procedure planning in instructional videos entails predicting an action sequence that transitions a given start state to a desired goal state. This task is particularly challenging due to two key sources of uncertainty: limited visual observations and an enormous decision space. The former results in multiple plausible plan variations due to missing intermediate visual states, while the latter complicates prediction by requiring selection from a large set of potential actions. Unlike prior work that addresses these issues implicitly, we propose an explicit solution. To mitigate the first challenge, we employ image generation models to synthesize diverse intermediate visual states using various text prompts, followed by a prompt selection module integrated within a diffusion model. To tackle the second challenge, we introduce a task-selective diffusion model that applies a task-specific mask to constrain the action space. As the effectiveness of this mask depends on accurate task classification, we further enhance visual representation by leveraging pre-trained vision-language models to generate action-aware, text-enriched multimodal embeddings. Extensive experiments on three benchmark datasets validate the superior performance of our proposed approach.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the A*STAR - Manufacturing, Trade, and Connectivity Programmatic Fund
Grant Reference no. : M23L7b0021
This research / project is supported by the A*STAR - Advanced Manufacturing and Engineering (AME) Programmatic Fund
Grant Reference no. : A18A2b0046
This research / project is supported by the Singapore Economic Development Board (EDB) - Space Technology Development Programme
Grant Reference no. : S22-19016-STDP
This research / project is supported by the National Natural Science Foundation of China - NA
Grant Reference no. : 62171343