Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networks

Page view(s)
118
Checked on Nov 01, 2024
Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networks
Title:
Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networks
Journal Title:
International Conference on Computer Vision (ICCV) 2023
DOI:
Keywords:
Publication Date:
08 October 2023
Citation:
Xu, Kaixin; Wang, Zhe; Geng, Xue; Wu, Min; Li, Xiaoli; Lin, Weisi
Abstract:
In this paper, we propose a novel layer-adaptive weight-pruning approach for Deep Neural Networks (DNNs) that addresses the challenge of optimizing the output distortion minimization while adhering to a target pruning ratio constraint. Our approach takes into account the collective influence of all layers to design a layer-adaptive pruning scheme. We discover and utilize a very important additivity property of output distortion caused by pruning weights on multiple layers. This property enables us to formulate the pruning as a combinatorial optimization problem and efficiently solve it through dynamic programming. By decomposing the problem into sub-problems, we achieve linear time complexity, making our optimization algorithm fast and feasible to run on CPUs. Our extensive experiments demonstrate the superiority of our approach over existing methods on the ImageNet and CIFAR-10 datasets. On CIFAR-10, our method achieves remarkable improvements, outperforming others by up to 1.0% for ResNet-32, 0.5% for VGG-16, and 0.7% for DenseNet-121 in terms of top-1 accuracy. On ImageNet, we achieve up to 4.7% and 4.6% higher top-1 accuracy compared to other methods for VGG-16 and ResNet-50, respectively. These results highlight the effectiveness and practicality of our approach for enhancing DNN performance through layer-adaptive weight pruning. Code will be available on https://github.com/Akimoto-Cris/RD_VIT_PRUNE.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the ASTAR - AI3 HTCO Seed Fund
Grant Reference no. : A1892b0026

This research / project is supported by the ASTAR - GAP Fund
Grant Reference no. : C211118009

This research / project is supported by the ASTAR - MTC Programmatic
Grant Reference no. : M23L7b0021
Description:
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
ISSN:
N.A
Files uploaded:

File Size Format Action
07352.pdf 632.32 KB PDF Request a copy