Haicheng Qu
ABSTRACT
Current filter pruning methods rely too much on pretrained weights and have many super parameters, resulting in obvious performance degradation and too long parameters adjustment time. In our research, we found that the cosine similarity distribution between filters can achieve stable in a few epochs during training. Therefore, a cluster pruning method named ECP(Early Cluster Pruning) based on the cosine similarity between filters in the early stage of training is proposed to compress the deep neural networks. First, in the early stage of training, the filters were clustered with a gradually increasing threshold, and then the reserved filters were selected randomly in each cluster. The pruned models could be obtained with only a few super parameters and a single training progress, leading to an obvious reduction in algorithmic complexity and large savings in training time. The experimental results on CIFAR-10 and CIFAR-100 datasets show that ECP method outperforms recent pruning methods in terms of model accuracy maintenance, training time, and model compression rate.
- Qi Fan, Deng-Ping Fan, Huazhu Fu, Chi-Keung Tang, Ling Shao, Yu-Wing Tai.2021. Group collaborative learning for co-salient object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 12288-12298. https://doi.org/10.1109/CVPR46437.2021.01211.Google Scholar
Cross Ref
- Wentong Li Wentong, Yijie Chen, Kaixuan Hu, Jianke Zhu. 2022. Oriented reppoints for aerial object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1829-1838. https://doi.org/10.1109/CVPR52688.2022.00187.Google ScholarCross Ref
- Chienyao Wang, Alexey Bochkovskiy, Hongyuan Mark Liao. 2022.YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[J]. arXiv preprint. https://doi.org/10.48550/arXiv.2207.02696.Google ScholarCross Ref
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby. 2020.An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint. https://doi.org/10.48550/arXiv.2207.02696.Google ScholarCross Ref
- Shanghua Gao, Mingming Cheng, Kai Zhao, Xinyu Zhang, Ming-Hsuan Yang,Philip Torr. 2019.Res2net: A new multi-scale backbone architecture. IEEE transactions on pattern analysis and machine intelligence, 43(2): 652-662. https://doi.org/10.1109/TPAMI.2019.2938758.Google ScholarDigital Library
- Guillaume Couairon, Asya Grechka, Jakob Verbeek, Holger Schwenk, Matthieu Cord. 2022.FlexIT: towards flexible semantic image translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 18270-18279. https://doi.org/10.1109/CVPR52688.2022.01773.Google ScholarCross Ref
- Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, Fang Wen. 2020. Cross-domain correspondence learning for exemplar-based image translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5143-5153. https://doi.org/10.1109/CVPR42600.2020.00519.Google ScholarCross Ref
- Mingwen Sha, Junhui Dai, Ran Wang, Jiandong Kuang, Wangmeng Zuo. 2022. CSHE: network pruning by using cluster similarity and matrix eigenvalues. International Journal of Machine Learning and Cybernetics, 13(2): 371-382. https://doi.org/10.1007/s13042-021-01411-8.Google ScholarCross Ref
- Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, Ling Shao. 2020. Hrank: Filter pruning using high-rank feature map. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,1529-1538. https://doi.org/10.1109/CVPR42600.2020.00160.Google ScholarCross Ref
- Babajide O.Ayinde, Tamer Inanc, Jacek M. Zurada. 2019. Redundant feature pruning for accelerated inference in deep neural networks. Neural Networks,118: 148-158. https://doi.org/10.1016/j.neunet.2019.04.021.Google ScholarDigital Library
- Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf. 2016. Pruning filters for efficient convnets. arXiv preprint. https://doi.org/10.48550/arXiv.1608.08710.Google ScholarCross Ref
- Mingbao Lin, Rongrong Ji, Bohong Chen, Fei Chao, Jianzhuang Liu, Wei Zeng, Yonghong Tian, Qi Tian. 2021. Training Compact CNNs for Image Classification using Dynamic-coded Filter Fusion. arXiv preprint.https://doi.org/10.48550/arXiv.2107.06916.Google ScholarCross Ref
- Mingbao Lin, Rongrong Ji, Yuxin Zhang, Baochang Zhang, Yongjian Wu, Yonghong Tian.2020. Channel pruning via automatic structure search. arXiv preprint. https://doi.org/10.48550/arXiv.2001.08565.Google ScholarCross Ref
- Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, Trevor Darrell. 2018. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270. https://doi.org/10.48550/arXiv.1810.05270.Google ScholarCross Ref
- Jonathan Frankle, Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint. https://doi.org/10.48550/arXiv.1803.03635.Google ScholarCross Ref
- Karen Simonyan, Andrew Zisserman.2009. Very deep convolutional networks for large-scale image recognition. arXiv preprint. https://doi.org/10.48550/arXiv.1409.1556.Google ScholarCross Ref
- Alex Krizhevsky. 2009. Learning multiple layers of features from tiny images. Master's thesis, University of Tront.Google Scholar
- Fanxu Meng, Hao Cheng, Ke Li, Huixiang Luo, Xiaowei Guo, Guangming Lu, Xing Sun.2020. Pruning filter in filter. Advances in Neural Information Processing Systems, 33: 17629-17640.Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun.2016. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition,770-778. https://doi.org/10.1109/cvpr.2016.90.Google ScholarCross Ref
- Mingbao Lin, Liujuan Cao, Yuxin Zhang, Ling Shao, Chia-Wen Lin, Rongrong Ji. 2022. Pruning Networks With Cross-Layer Ranking & k-Reciprocal Nearest Filters. IEEE Transactions on Neural Networks and Learning Systems, 2022. https://doi.org/10.1109/TNNLS.2022.3156047.Google ScholarCross Ref
- Huan Wang, Can Qin, Yulun Zhang, Yun Fu. 2020.Neural pruning via growing regularization. arXiv preprint. https://doi.org/10.48550/arXiv.2012.09243.Google ScholarCross Ref
- Xinyu Liu, Baopu Li, Zhen Chen, Yixuan Yuan. 2021. Exploring Gradient Flow Based Saliency for DNN Model Compression. Proceedings of the 29th ACM International Conference on Multimedia, 3238-3246. https://dl.acm.org/doi/10.1145/3474085.3475474.Google ScholarDigital Library
- Rishabh Tiwari, Udbhav Bamba, Arnav Chavan, Deepak K. Gupta. 2021.Chipnet: Budget-aware pruning with heaviside continuous approximations. arXiv preprint. https://doi.org/10.48550/arXiv.2102.07156.Google ScholarCross Ref
- Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, Yi Yang.2018. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint. https://doi.org/10.48550/arXiv.1808.06866.Google ScholarCross Ref
- Fanxu Meng, Hao Cheng, Ke Li, Zhixin Xu, Rongrong Ji, Xing Sun, Guangming Lu.2020. Filter grafting for deep neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6599-6607. https://doi.org/10.1109/CVPR42600.2020.00663.Google ScholarCross Ref
Index Terms
- Pruning Networks Using Filters Similarity Stability
Recommendations
Filter Pruning via Similarity Clustering for Deep Convolutional Neural Networks
Neural Information ProcessingAbstractNetwork pruning is a technique to obtain a smaller lightweight model by removing the redundant structure from pre-trained models. However, existing methods are mainly based on the importance of filters in the whole network. Unlike previous methods,...
Linear Substitution Pruning: Consider All Filters Together
ICCAI '22: Proceedings of the 8th International Conference on Computing and Artificial IntelligenceFilter (neuron) pruning is a neural network compression approach. In previous work, the importance of each filter is generally considered individually or pairwisely. This paper shows that considering the linear relation among all filters can help us ...
Filter pruning - deeper layers need fewer filters
Model pruning aims to reduce the parameter amount of deep neural networks while retaining the performance. Existing strategies often treat all layers equally and all layers simply share the same pruning rate. However, it is observed from our experiments ...
Comments