Skip to main content
SpringerLink
Log in

Learning Phase Mask for Privacy-Preserving Passive Depth Estimation

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13667))

Included in the following conference series:

Abstract

With over a billion sold each year, cameras are not only becoming ubiquitous, but are driving progress in a wide range of domains such as mixed reality, robotics, and more. However, severe concerns regarding the privacy implications of camera-based solutions currently limit the range of environments where cameras can be deployed. The key question we address is: Can cameras be enhanced with a scalable solution to preserve users’ privacy without degrading their machine intelligence capabilities? Our solution is a novel end-to-end adversarial learning pipeline in which a phase mask placed at the aperture plane of a camera is jointly optimized with respect to privacy and utility objectives. We conduct an extensive design space analysis to determine operating points with desirable privacy-utility tradeoffs that are also amenable to sensor fabrication and real-world constraints. We demonstrate the first working prototype that enables passive depth estimation while inhibiting face identification.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Alhashim, I., Wonka, P.: High quality monocular depth estimation via transfer learning. arXiv preprint arXiv:1812.11941 (2018)

  2. Alvi, M., Zisserman, A., Nellåker, C.: Turning a blind eye: explicit removal of biases and variation from deep neural network embeddings. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11129, pp. 556–572. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11009-3_34

    Chapter  Google Scholar 

  3. Beach, S., Schulz, R., Downs, J., Matthews, J., Barron, B., Seelman, K.: Disability, age, and informational privacy attitudes in quality of life technology applications: results from a national web survey. ACM Trans. Access. Comput. 2(1), 5 (2009)

    Article  Google Scholar 

  4. Boominathan, V., Adams, J.K., Robinson, J.T., Veeraraghavan, A.: PhlatCam: designed phase-mask based thin lensless camera. IEEE Trans. Pattern Anal. Mach. Intell. 42(7), 1618–1629 (2020)

    Article  Google Scholar 

  5. Cao, Q., Shen, L., Xie, W., Parkhi, O.M., Zisserman, A.: Vggface2: a dataset for recognising faces across pose and age. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 67–74 (2018)

    Google Scholar 

  6. Carreira, J., Zisserman, A.: Quo vadis, action recognition? A new model and the kinetics dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308 (2017)

    Google Scholar 

  7. Chakrabarti, A.: Learning sensor multiplexing design through back-propagation. In: 30th Conference on Advances in Neural Information Processing Systems, pp. 3081–3089 (2016)

    Google Scholar 

  8. Chang, J., Wetzstein, G.: Deep optics for monocular depth estimation and 3D object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 10193–10202 (2019)

    Google Scholar 

  9. Chen, J., Konrad, J., Ishwar, P.: VGAN-based image representation learning for privacy-preserving facial expression recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1570–1579 (2018)

    Google Scholar 

  10. Chhabra, S., Singh, R., Vatsa, M., Gupta, G.: Anonymizing k-facial attributes via adversarial perturbations. arXiv preprint arXiv:1805.09380 (2018)

  11. Dai, J., Wu, J., Saghafi, B., Konrad, J., Ishwar, P.: Towards privacy-preserving activity recognition using extremely low temporal and spatial resolution cameras. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 68–76 (2015)

    Google Scholar 

  12. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009, pp. 248–255 (2009)

    Google Scholar 

  13. Dosovitskiy, A., Brox, T.: Generating images with perceptual similarity metrics based on deep networks. In: Advances in Neural Information Processing Systems, pp. 658–666 (2016)

    Google Scholar 

  14. Dwork, C.: Differential privacy: a survey of results. In: International Conference on Theory and Applications of Models of Computation, pp. 1–19 (2008)

    Google Scholar 

  15. Erdélyi, A., Barát, T., Valet, P., Winkler, T., Rinner, B.: Adaptive cartooning for privacy protection in camera networks. In: 2014 11th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 44–49 (2014)

    Google Scholar 

  16. Fu, H., Gong, M., Wang, C., Batmanghelich, K., Tao, D.: Deep ordinal regression network for monocular depth estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2002–2011 (2018)

    Google Scholar 

  17. Goodman, J.W.: Introduction to Fourier Optics. Roberts and Company Publishers, Englewood (2005)

    Google Scholar 

  18. Haim, H., Elmalem, S., Giryes, R., Bronstein, A.M., Marom, E.: Depth estimation from a single image using deep learned phase coded mask. IEEE Trans. Comput. Imag. 4(3), 298–310 (2018)

    Article  Google Scholar 

  19. He, L., Wang, G., Hu, Z.: Learning depth from single images with deep neural network embedding focal length. IEEE Trans. Image Process. 27(9), 4676–4689 (2018)

    Article  MathSciNet  Google Scholar 

  20. Hinojosa, C., Niebles, J.C., Arguello, H.: Learning privacy-preserving optics for human pose estimation. In: Proceedings of the IEEE/CVF International Conference

    Google Scholar 

  21. Hinojosa, C., Niebles, J.C., Arguello, H.: Learning privacy-preserving optics for human pose estimation. In: Proceedings of the IEEE/CVF International Conference

    Google Scholar 

  22. Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Tech. Rep. 07–49, University of Massachusetts, Amherst, October 2007

    Google Scholar 

  23. Jeong, Y., Yoo, D.H., Cho, J., Lee, B.: Optic design and image processing considering angle of incidence via end-to-end optimization method. Ultra-High-Defi. Imag Syst. II 10943, 109430U (2019)

    Google Scholar 

  24. Jia, S., Lansdall-Welfare, T., Cristianini, N.: Right for the right reason: training agnostic networks. In: International Symposium on Intelligent Data Analysis, pp. 164–174 (2018)

    Google Scholar 

  25. Kim, B., Kim, H., Kim, K., Kim, S., Kim, J.: Learning not to learn: training deep neural networks with biased data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9012–9020 (2019)

    Google Scholar 

  26. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  27. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, vol. 2 (2017)

    Google Scholar 

  28. Liu, J., Shahroudy, A., Perez, M.L., Wang, G., Duan, L.Y., Chichung, A.K.: Ntu RGB+ D 120: a large-scale benchmark for 3D human activity understanding. IEEE Trans. Pattern Anal. Mach. Intell. (2019)

    Google Scholar 

  29. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014)

    Google Scholar 

  30. Metzler, C.A., Ikoma, H., Peng, Y., Wetzstein, G.: Deep optics for single-shot high-dynamic-range imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1375–1385 (2020)

    Google Scholar 

  31. Mirjalili, V., Raschka, S., Ross, A.: Gender privacy: an ensemble of semi adversarial networks for confounding arbitrary gender classifiers. In: 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS), pp. 1–10 (2018)

    Google Scholar 

  32. Mirjalili, V., Raschka, S., Ross, A.: Flowsan: privacy-enhancaing semi-adversarial networks to confound arbitrary face-based gender classifiers. IEEE Access 7, 99735–99745 (2019)

    Article  Google Scholar 

  33. Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from rgbd images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7576, pp. 746–760. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33715-4_54

    Chapter  Google Scholar 

  34. Nawaz, T., Rinner, B., Ferryman, J.: User-centric, embedded vision-based human monitoring: a concept and a healthcare use case. In: Proceedings of the 10th International Conference on Distributed Smart Camera, pp. 25–30 (2016)

    Google Scholar 

  35. Neustaedter, C.G., Greenberg, S.: Balancing privacy and awareness in home media spaces. In: Citeseer (2003)

    Google Scholar 

  36. Nguyen Canh, T., Nagahara, H.: Deep compressive sensing for visual privacy protection in Flatcam imaging. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0–0 (2019)

    Google Scholar 

  37. Padilla-López, J.R., Chaaraoui, A.A., Flórez-Revuelta, F.: Visual privacy protection methods: A survey. Expert Syst. Appl. 42(9), 4177–4195 (2015)

    Article  Google Scholar 

  38. Phan, B., Mannan, F., Heide, F.: Adversarial imaging pipelines. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16051–16061 (2021)

    Google Scholar 

  39. Pittaluga, F., Koppal, S., Chakrabarti, A.: Learning privacy preserving encodings through adversarial training. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 791–799 (2019)

    Google Scholar 

  40. Pittaluga, F., Koppal, S.J.: Privacy preserving optics for miniature vision sensors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 314–324 (2015)

    Google Scholar 

  41. Pittaluga, F., Koppal, S.J.: Pre-capture privacy for small vision sensors. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2215–2226 (2016)

    Article  Google Scholar 

  42. Pittaluga, F., Zivkovic, A., Koppal, S.J.: Sensor-level privacy for thermal cameras. In: 2016 IEEE International Conference on Computational Photography (ICCP), pp. 1–12 (2016)

    Google Scholar 

  43. Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. ArXiv preprint (2021)

    Google Scholar 

  44. Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., Koltun, V.: Towards robust monocular depth estimation: mixing datasets for zero-shot cross-dataset transfer. IEEE Trans. Pattern Anal. Mach. Intell.44, 1623–1637 (2020)

    Google Scholar 

  45. Sattar, H., Krombholz, K., Pons-Moll, G., Fritz, M.: Shape evasion: Preventing body shape inference of multi-stage approaches. arXiv preprint arXiv:1905.11503 (2019)

  46. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  47. Sitzmann, V., et al.: End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Trans. Graph. 37(4), 1–13 (2018)

    Article  Google Scholar 

  48. Sun, Q., Tseng, E., Fu, Q., Heidrich, W., Heide, F.: Learning rank-1 diffractive optics for single-shot high dynamic range imaging. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 1386–1396 (2020)

    Google Scholar 

  49. Tan, J., et al.: Canopic: Pre-digital privacy-enhancing encodings for computer vision. In: 2020 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6 (2020)

    Google Scholar 

  50. Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114 (2019)

    Google Scholar 

  51. Tseng, E., et al.: Differentiable compound optics and processing pipeline optimization for end-to-end camera design. ACM Trans. Graph. 40(2), 1–19 (2021)

    Article  Google Scholar 

  52. Wang, H., et al.:Off-axis holography with uniform illumination via 3D printed diffractive optical elements. Adv. Opt. Mater.7(12), 1900068 (2019)

    Google Scholar 

  53. Wang, L., Zhang, J., Wang, O., Lin, Z., Lu, H.: SDC-depth: semantic divide-and-conquer network for monocular depth estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 541–550 (2020)

    Google Scholar 

  54. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  55. Wang, Z.W., Vineet, V., Pittaluga, F., Sinha, S.N., Cossairt, O., Bing Kang, S.: Privacy-preserving action recognition using coded aperture videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2019)

    Google Scholar 

  56. Winkler, T., Erdélyi, A., Rinner, B.: TrustEYE. M4: protecting the sensor-not the camera. In: 2014 11th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 159–164 (2014)

    Google Scholar 

  57. Wu, Y., Boominathan, V., Chen, H., Sankaranarayanan, A., Veeraraghavan, A.: Phasecam3d-learning phase masks for passive single view depth estimation. In: 2019 IEEE International Conference on Computational Photoagraphy (ICCP), pp. 1–12. IEEE (2019)

    Google Scholar 

  58. Wu, Y., Yang, F., Ling, H.: Privacy-protective-GAN for face de-identification. arXiv preprint arXiv:1806.08906 (2018)

  59. Xia, Z., Sullivan, P., Chakrabarti, A.: Generating and exploiting probabilistic monocular depth estimates. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 65–74 (2020)

    Google Scholar 

  60. Xian, K., et al.: Monocular relative depth perception with web stereo data supervision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 311–320 (2018)

    Google Scholar 

  61. Xiao, T., Tsai, Y.H., Sohn, K., Chandraker, M., Yang, M.H.: Adversarial learning of privacy-preserving and task-oriented representations. In: Proceedings of the AAAI Conference on Artificial Intelligence (2020)

    Google Scholar 

  62. Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492–1500 (2017)

    Google Scholar 

  63. Yang, Q., Liu, Y., Chen, T., Tong, Y.: Federated machine learning: Concept and applications. ACM Trans. Intell. Syst. Technol. 10(2), 1–19 (2019)

    Article  Google Scholar 

  64. Yin, W., Liu, Y., Shen, C., Yan, Y.: Enforcing geometric constraints of virtual normal for depth prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5684–5693 (2019)

    Google Scholar 

  65. Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Proacess. Lett. 23(10), 1499–1503 (2016)

    Article  Google Scholar 

  66. Zhuang, Z., Bradtmiller, B.: Head-and-face anthropometric survey of us respirator users. J. Occup. Environ. Hyg. 2(11), 567–576 (2005)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. NEC Laboratories America, Princeton, USA

    Giovanni Milione, Yi-Hsuan Tsai, Xiang Yu, Manmohan Chandraker & Francesco Pittaluga

  2. Rice University, Houston, USA

    Zaid Tasneem & Ashok Veeraraghavan

  3. University of California San Diego, San Diego, USA

    Manmohan Chandraker

Authors
  1. Zaid Tasneem
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giovanni Milione
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yi-Hsuan Tsai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiang Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ashok Veeraraghavan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Manmohan Chandraker
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Francesco Pittaluga
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Francesco Pittaluga .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 2238 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Tasneem, Z. et al. (2022). Learning Phase Mask for Privacy-Preserving Passive Depth Estimation. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13667. Springer, Cham. https://doi.org/10.1007/978-3-031-20071-7_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20071-7_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20070-0

  • Online ISBN: 978-3-031-20071-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation