Cao, T., Yu, W., Gao, Y., Liu, C., Yan, S., & Goh, W. L. (2023, May 21). RRAM-PoolFormer: A Resistive Memristor-based PoolFormer Modeling and Training Framework for Edge-AI Applications. 2023 IEEE International Symposium on Circuits and Systems (ISCAS). https://doi.org/10.1109/iscas46773.2023.10181612
Abstract:
PoolFormer is a type of neural network architecture that is abstracted from Transformer where the computationally heavy token mixer module is replaced with simple pooling function. This paper presents a memristor-based PoolFormer modeling and training framework for edge-AI applications. To fit for implementation on resistive crossbar array, original PoolFormer structure is further optimized by replacing normalization operation with hardware friendly scaling operation. In addition, the non-idealities of RRAM crossbar from device to array level as well as peripheral readout circuits are also included. By incorporating these elements under a single framework for network training, their impact to the network performance can be effectively mitigated. This framework is implemented in a combination of Python and PyTorch. A 16-block PoolFormer network is designed and optimized for CIFAR-10 image classification tasks using measured 64×64 RRAM crossbar array results. The total network weights are only 0.26M, which is at least one order of magnitude smaller than that of the conventional DNN implementation. When compared to the ideal model with FP64 weight bit-length, 85.86% inference accuracy is reached with only 4-level weight resolutions and less than 4% accuracy loss.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the A*STAR - Nanosystems at the Edge (WP03)
Grant Reference no. : A18A1b0055