Wu, Y.-H., Zhang, S.-C., Liu, Y., Zhang, L., Zhan, X., Zhou, D., Feng, J., Cheng, M.-M., & Zhen, L. (2025). Low-Resolution Self-Attention for Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(9), 8180–8192. https://doi.org/10.1109/tpami.2025.3577035
Abstract:
Semantic segmentation tasks naturally require high-resolution information for pixel-wise segmentation and global context information for class prediction. While existing vision transformers demonstrate promising performance, they often utilize high-resolution context modeling, resulting in a computational bottleneck. In this work, we challenge conventional wisdom and introduce the Low-Resolution Self-Attention (LRSA) mechanism to capture global context at a significantly reduced computational cost, i.e., FLOPs. Our approach involves computing self-attention in a fixed low-resolution space, regardless of the input image’s resolution, with additional 3×3 depth-wise convolutions to capture fine details in the high-resolution space. We demonstrate the effectiveness of our LRSA approach by building the LRFormer, a vision transformer with an encoder-decoder structure. Extensive experiments on the ADE20 K, COCO-Stuff, and CityScapes datasets demonstrate that LRFormer outperforms state-of-the-art models.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the A*STAR - Career Development Fund
Grant Reference no. : C233312006
This research / project is supported by the National Research Foundation - AI Singapore Programme
Grant Reference no. : AISG2-GC 2023-007