Nandan Kumar Jha
Sparsh Mittal
Skip Abstract Section
Abstract
The remarkable predictive performance of deep neural networks (DNNs) has led to their adoption in service domains of unprecedented scale and scope. However, the widespread adoption and growing commercialization of DNNs have underscored the importance of intellectual property (IP) protection. Devising techniques to ensure IP protection has become necessary due to the increasing trend of outsourcing the DNN computations on the untrusted accelerators in cloud-based services. The design methodologies and hyper-parameters of DNNs are crucial information, and leaking them may cause massive economic loss to the organization. Furthermore, the knowledge of DNN’s architecture can increase the success probability of an adversarial attack where an adversary perturbs the inputs and alters the prediction.
In this work, we devise a two-stage attack methodology “DeepPeep,” which exploits the distinctive characteristics of design methodologies to reverse-engineer the architecture of building blocks in compact DNNs. We show the efficacy of “DeepPeep” on P100 and P4000 GPUs. Additionally, we propose intelligent design maneuvering strategies for thwarting IP theft through the DeepPeep attack and proposed “Secure MobileNet-V1.” Interestingly, compared to vanilla MobileNet-V1, secure MobileNet-V1 provides a significant reduction in inference latency (≈60%) and improvement in predictive performance (≈2%) with very low memory and computation overheads.
- Jeremy Howard. 2019. imagenette. Retrieved from https://github.com/fastai/imagenette.Google Scholar
- Hal Conick. 2017. The past, present and future of AI in marketing. Marketing News 51, 1 (2017), 26--35.Google Scholar
- CUDA Nvidia. 2008. Cublas library. NVIDIA Corporation, Santa Clara, California 15, 27 (2008), 31.Google Scholar
- Dario Amodei and Danny Hernandez. 2018. AI and Compute. Retrieved from https://openai.com/blog/ai-and-compute/.Google Scholar
- Robert A. Bridges, Neena Imam, and Tiffany M. Mintz. 2016. Understanding GPU power: A survey of profiling, modeling, and simulation methods. ACM Comput. Surv. 49, 3 (2016), 1--27.Google ScholarDigital Library
- Victor Costan and Srinivas Devadas. 2016. Intel SGX explained. IACR Cryptol. ePrint Arch. 2016, 86 (2016), 1--118.Google Scholar
- Victor Costan, Ilia A. Lebedev, and Srinivas Devadas. 2016. Sanctum: Minimal hardware extensions for strong software isolation. In Proceedings of the USENIX Security Symposium.Google Scholar
- Alessandro Del Sole. 2018. Introducing Microsoft cognitive services. In Microsoft Computer Vision APIs Distilled. Springer, 1--4.Google Scholar
- Amir Gholami, Kiseok Kwon, Bichen Wu, Zizheng Tai, Xiangyu Yue, Peter H. Jin, Sicheng Zhao, and Kurt Keutzer. 2018. SqueezeNext: Hardware-aware neural network design. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.Google ScholarCross Ref
- Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. Arxiv Preprint Arxiv:1412.6572 (2014).Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16).Google ScholarCross Ref
- Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017).Google Scholar
- Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7132--7141.Google ScholarCross Ref
- W. Hua, Z. Zhang, and G. E. Suh. 2018. Reverse engineering convolutional neural networks through side-channel information leaks. In Proceedings of the Design Automation Conference. 1--6.Google Scholar
- G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2261--2269.Google Scholar
- Forrest N. Iandola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and MB model size. CoRR abs/1602.07360 (2016).Google Scholar
- N. K. Jha, S. Mittal, and G. Mattela. 2019. The ramifications of making deep neural networks compact. In Proceedings of the 32nd International Conference on VLSI Design and 18th International Conference on Embedded Systems (VLSID’19). 215--220.Google Scholar
- Nandan Kumar Jha, Rajat Saini, Subhrajit Nag, and Sparsh Mittal. 2020. E2GC: Energy-efficient group convolution in deep neural networks. In Proceedings of the 33rd International Conference on VLSI Design and 19th International Conference on Embedded Systems (VLSID’20). IEEE, 155--160.Google ScholarCross Ref
- Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the International Conference on Multimedia (MM’14). 675--678.Google ScholarDigital Library
- Alex Kaplunovich and Yelena Yesha. 2017. Cloud big data decision support system for machine learning on AWS: Analytics of analytics. In Proceedings of the IEEE International Conference on Big Data (Big Data’17). IEEE, 3508--3516.Google ScholarCross Ref
- B. Kehoe, S. Patil, P. Abbeel, and K. Goldberg. 2015. A survey of research on cloud robotics and automation. IEEE Trans. Automat. Sci. Eng. 12, 2 (Apr. 2015), 398--409.Google ScholarCross Ref
- Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. 2012. CIFAR-100 (Canadian Institute for Advanced Research). (2012). Retrieved from http://www.cs.toronto.edu/ kriz/cifar.html.Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the Conference on Neural Information Processing Systems. 1097--1105.Google Scholar
- Liangzhen Lai, Naveen Suda, and Vikas Chandra. 2018. Not all ops are created equal! arXiv preprint arXiv:1801.04326 (2018).Google Scholar
- Wei Li, Kai Liu, Lin Yan, Fei Cheng, YunQiu Lv, and LiZhe Zhang. 2019. FRD-CNN: Object detection based on small-scale convolutional neural networks and feature reuse. Sci. Rep. 9, 1 (2019), 1--12.Google Scholar
- Y. Li and A. Pedram. 2017. CATERPILLAR: Coarse grain reconfigurable architecture for accelerating the training of deep neural networks. In Proceedings of the IEEE International Conference on Application-specific Systems. 1--10.Google Scholar
- Qi Liu, Tao Liu, Zihao Liu, Yanzhi Wang, Yier Jin, and Wujie Wen. 2018. Security analysis and enhancement of model compressed deep learning systems under adversarial attacks. In Proceedings of the Asia and South Pacific Design Automation Conference. 721--726.Google ScholarCross Ref
- T. Liu, W. Wen, and Y. Jin. 2018. SIN2: Stealth infection on neural network–A low-cost agile neural Trojan attack methodology. In Proceedings of the IEEE International Symposium on Hardware Oriented Security and Trust. 227--230.Google Scholar
- Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. 2016. Delving into transferable adversarial examples and black-box attacks. Arxiv Preprint Arxiv:1611.02770 (2016).Google Scholar
- Yunhui Long, Vincent Bindschaedler, Lei Wang, Diyue Bu, Xiaofeng Wang, Haixu Tang, Carl A. Gunter, and Kai Chen. 2018. Understanding membership inferences on well-generalized learning models. CoRR abs/1802.04889 (2018).Google Scholar
- Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. 2018. ShuffleNet V2: Practical guidelines for efficient CNN architecture design. In Proceedings of the European Conference on Computer Vision.Google ScholarCross Ref
- Asit K. Mishra, Joseph L. Hellerstein, Walfredo Cirne, and Chita R. Das. 2010. Towards characterizing cloud backend workloads: Insights from Google compute clusters. ACM SIGMETRICS Perf. Eval. Rev. 37, 4 (2010), 34--41.Google ScholarDigital Library
- Sparsh Mittal, S. B. Abhinaya, Manish Reddy, and Irfan Ali. 2018. A survey of techniques for improving security of GPUs. Hardw. Syst. Secur. J. 2, 3 (2018), 266--285.Google ScholarCross Ref
- Seong Joon Oh, Max Augustin, Mario Fritz, and Bernt Schiele. 2018. Towards reverse-engineering black-box neural networks. In Proceedings of the International Conference on Learning Representations.Google Scholar
- Bita Darvish Rouhani, Huili Chen, and Farinaz Koushanfar. 2019. DeepSigns: A generic watermarking framework for IP protection of deep learning models. In Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems.Google Scholar
- Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. Imagenet large scale visual recognition challenge. Int. J. Comput. Vision 115, 3 (2015), 211--252.Google ScholarDigital Library
- M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen. 2018. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4510--4520.Google Scholar
- R. Shokri, M. Stronati, C. Song, and V. Shmatikov. 2017. Membership inference attacks against machine learning models. In Proceedings of the IEEE Symposium on Security and Privacy (SP’17).Google Scholar
- Ion Stoica, Dawn Song, Raluca Ada Popa, David Patterson, Michael W. Mahoney, Randy Katz, Anthony D. Joseph, Michael Jordan, Joseph M. Hellerstein, Joseph E. Gonzalez, et al. 2017. A Berkeley view of systems challenges for AI. Arxiv Preprint Arxiv:1712.05855 (2017).Google Scholar
- Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google ScholarCross Ref
- Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google ScholarCross Ref
- Florian Tramer and Dan Boneh. 2019. Slalom: Fast, verifiable and private execution of neural networks in trusted hardware. In Proceedings of the International Conference on Learning Representations.Google Scholar
- Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing machine learning models via prediction APIs. In Proceedings of the 25th USENIX Security Symposium (SEC’16). 601--618.Google ScholarDigital Library
- B. Wang and N. Z. Gong. 2018. Stealing hyperparameters in machine learning. In Proceedings of the IEEE Symposium on Security and Privacy (SP’18).Google Scholar
- Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska. 2018. Superneurons: Dynamic GPU memory management for training deep neural networks. In Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 41--53.Google ScholarDigital Library
- Lilian Weng. 2018. Attention? Attention! Retrieved from https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html.Google Scholar
- Bichen Wu, Forrest Iandola, Peter H. Jin, and Kurt Keutzer. 2017. Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 129--137.Google ScholarCross Ref
- Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5987--5995.Google ScholarCross Ref
- Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He. 2019. Exploring randomly wired neural networks for image recognition. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’19).Google ScholarCross Ref
- Keyulu Xu, Jingling Li, Mozhi Zhang, Simon S. Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. 2020. What can neural networks reason about? In Proceedings of the International Conference on Learning Representations.Google Scholar
- Mengjia Yan, Christopher Fletcher, and Josep Torrellas. 2020. Cache telepathy: Leveraging shared resource attacks to learn DNN architectures. In Proceedings of the 29th USENIX Security Symposium (USENIX Security’20).Google Scholar
- Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. In Proceedings of the British Machine Vision Conference.Google ScholarCross Ref
- Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. 2018. ShuffleNet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google ScholarCross Ref
- Shijun Zhao, Qianying Zhang, Guangyao Hu, Yu Qin, and Dengguo Feng. 2014. Providing root of trust for ARM trustzone using on-chip SRAM. In Proceedings of the 4th International Workshop on Trustworthy Embedded Devices (TrustED’14). 25--36.Google ScholarDigital Library
- Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. 2018. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
Index Terms
- DeepPeep: Exploiting Design Ramifications to Decipher the Architecture of Compact DNNs
Recommendations
New passive and active attacks on deep neural networks in medical applications
ICCAD '20: Proceedings of the 39th International Conference on Computer-Aided DesignSecurity of deep neural network (DNN) inference engines, i.e., trained DNN models on various platforms, has become one of the biggest challenges in deploying artificial intelligence in domains where privacy, safety, and reliability are of paramount ...
Is approximation universally defensive against adversarial attacks in deep neural networks?
DATE '22: Proceedings of the 2022 Conference & Exhibition on Design, Automation & Test in EuropeApproximate computing is known for its effectiveness in improvising the energy efficiency of deep neural network (DNN) accelerators at the cost of slight accuracy loss. Very recently, the inexact nature of approximate components, such as approximate ...
Are Timing-Based Side-Channel Attacks Feasible in Shared, Modern Computing Hardware?
This article describes how there exist various vulnerabilities in computing hardware that adversaries can exploit to mount attacks against the users of such hardware. Microarchitectural attacks, the result of these vulnerabilities, take advantage of ...
Comments