Skip to main content
SpringerLink
Log in

Approximate Differentiable Rendering with Algebraic Surfaces

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13692))

Included in the following conference series:

Abstract

Differentiable renderers provide a direct mathematical link between an object’s 3D representation and images of that object. In this work, we develop an approximate differentiable renderer for a compact, interpretable representation, which we call Fuzzy Metaballs. Our approximate renderer focuses on rendering shapes via depth maps and silhouettes. It sacrifices fidelity for utility, producing fast runtimes and high-quality gradient information that can be used to solve vision tasks. Compared to mesh-based differentiable renderers, our method has forward passes that are 5x faster and backwards passes that are 30x faster. The depth maps and silhouette images generated by our method are smooth and defined everywhere. In our evaluation of differentiable renderers for pose estimation, we show that our method is the only one comparable to classic techniques. In shape from silhouette, our method performs well using only gradient descent and a per-pixel loss, without any surrogate losses or regularization. These reconstructions work well even on natural video sequences with segmentation artifacts. Project page: https://leonidk.github.io/fuzzy-metaballs

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Unbiased gradient estimation for differentiable surface splatting via Poisson sampling. In: European Conference on Computer Vision (ECCV) (2022)

    Google Scholar 

  2. Adams, B., Lenaert, T., Dutré, P.: Particle splatting: interactive rendering of particle-based simulation data. Report CW 453, KU Leuven, July 2006. https://www.cs.kuleuven.be/publicaties/rapporten/cw/CW453.abs.html

  3. Agin, G.J.: Representation and Description of Curved Objects. Ph.D. thesis, Stanford University, CA, USA (1972)

    Google Scholar 

  4. Bangaru, S., Li, T.M., Durand, F.: Unbiased warped-area sampling for differentiable rendering. ACM Trans. Graph. 39(6), 245:1–245:18 (2020)

    Google Scholar 

  5. Bell, C.G., Fujisaki, H., Heinz, J.M., Stevens, K.N., House, A.S.: Reduction of speech spectra by analysis-by-synthesis techniques. J. Acoust. Soc. Am. 33(12), 1725–1736 (1961). https://doi.org/10.1121/1.1908556

  6. Blinn, J.F.: A generalization of algebraic surface drawing. ACM Trans. Graph. 1(3), 235–256 (1982). https://doi.org/10.1145/357306.357310

    Article  Google Scholar 

  7. Blinn, J.F.: How to solve a cubic equation, part 5: back to numerics. IEEE Comput. Graph. Appl. 27(3), 78–89 (2007). https://doi.org/10.1109/MCG.2007.60

    Article  Google Scholar 

  8. Bradbury, J., et al.: JAX: composable transformations of Python+NumPy programs (2018). https://github.com/google/jax

  9. Brubaker, M., Punjani, A., Fleet, D.: Building proteins in a day. CVPR (2015). https://doi.org/10.1109/cvpr.2015.7298929

  10. Chen, W., et al.: Learning to predict 3D objects with an interpolation-based differentiable renderer. Adv. Neural Inf. Process. Syst. 32 (2019)

    Google Scholar 

  11. Cheung, K.M.G., Baker, S., Kanade, T.: Shape-from-silhouette across time part I: theory and algorithms. Int. J. Comput. Vis. 62(3), 221–247 (2005). https://doi.org/10.1007/s11263-005-4881-5

  12. Cole, F., Genova, K., Sud, A., Vlasic, D., Zhang, Z.: Differentiable surface rendering via non-differentiable sampling (2021)

    Google Scholar 

  13. Dempster, A., Laird, N., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. Royal Stat. Soc. (B) 39(1), 1–38 (1977). https://doi.org/10.2307/2984875

    Article  MathSciNet  MATH  Google Scholar 

  14. Eckart, B., Kim, K., Kautz, J.: HGMR: hierarchical gaussian mixtures for adaptive 3D registration. In: ECCV 2018, pp. 730–746 (2018)

    Google Scholar 

  15. Eckart, B., Kim, K., Troccoli, A., Kelly, A., Kautz, J.: MLMD: maximum likelihood mixture decoupling for fast and accurate point cloud registration. In: 3DV, pp. 241–249 (2015). https://doi.org/10.1109/3DV.2015.34

  16. Eckart, B., Kim, K., Troccoli, A., Kelly, A., Kautz, J.: Accelerated generative models for 3D point cloud data. In: CVPR, pp. 5497–5505 (2016). https://doi.org/10.1109/CVPR.2016.593

  17. Enderton, E., Sintorn, E., Shirley, P., Luebke, D.: Stochastic transparency. In: I3D 2010: Proceedings of the 2010 Symposium on Interactive 3D Graphics and Games, pp. 157–164. New York, NY, USA (2010)

    Google Scholar 

  18. Garland, M., Heckbert, P.S.: Surface simplification using quadric error metrics. In: SIGGRAPH, pp. 209–216 (1997). https://doi.org/10.1145/258734.258849

  19. Genova, K., Cole, F., Sud, A., Sarna, A., Funkhouser, T.: Local deep implicit functions for 3D shape (2020)

    Google Scholar 

  20. Gourmel, O., Pajot, A., Paulin, M., Barthe, L., Poulin, P.: Fitted BVH for fast raytracing of metaballs. Comput. Graph. Forum 3, 7–288 (2010)

    Google Scholar 

  21. He, K., Gkioxari, G., Dollár, P., Girshick, R.B.: Mask R-CNN. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988 (2017)

    Google Scholar 

  22. Heckbert, P.S.: Fun with gaussians. In: SIGGRAPH 1986 Advanced Image Processing Seminar Notes (1986)

    Google Scholar 

  23. Hertz, A., Hanocka, R., Giryes, R., Cohen-Or, D.: PointGMM: a neural GMM network for point clouds. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2020)

    Google Scholar 

  24. Horvath, R.: Image-Space Metaballs Using Deep Learning. Master’s thesis, Faculty of Informatics, TU Wien, July 2019. https://www.cg.tuwien.ac.at/research/publications/2019/horvath-2018-ism/

  25. Huang, H., Ye, H., Sun, Y., Liu, M.: GMMLoc: structure consistent visual localization with gaussian mixture models. IEEE Robot. Autom. Lett. 5(4), 5043–5050 (2020). https://doi.org/10.1109/LRA.2020.3005130

    Article  Google Scholar 

  26. Insafutdinov, E., Dosovitskiy, A.: Unsupervised learning of shape and pose with differentiable point clouds (2018)

    Google Scholar 

  27. Kaasalainen, M., Torppa, J.: Optimization methods for asteroid lightcurve inversion. Icarus 153(1), 24–36 (2001)

    Article  Google Scholar 

  28. Kato, H., Ushiku, Y., Harada, T.: Neural 3D mesh renderer (2017)

    Google Scholar 

  29. Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Sheffer, A., Polthier, K. (eds.) Symposium on Geometry Processing. The Eurographics Association (2006). https://doi.org/10.2312/SGP/SGP06/061-070

  30. Keselman, L., Hebert, M.: Direct fitting of gaussian mixture models. In: 2019 16th Conference on Computer and Robot Vision (CRV), pp. 25–32 (2019). https://doi.org/10.1109/CRV.2019.00012

  31. Keselman, L., Woodfill, J.I., Grunnet-Jepsen, A., Bhowmik, A.: Intel realsense stereoscopic depth cameras. CoRR abs/1705.05548 (2017). arxiv:1705.05548

  32. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (Poster) (2015). arxiv:1412.6980

  33. Laine, S., Hellsten, J., Karras, T., Seol, Y., Lehtinen, J., Aila, T.: Modular primitives for high-performance differentiable rendering. ACM Trans. Graph. 39(6), 1–14 (2020)

    Google Scholar 

  34. Lassner, C., Zollhöfer, M.: Pulsar: efficient sphere-based neural rendering. arXiv:2004.07484 (2020)

  35. Levoy, M., Gerth, J., Curless, B., Pull, K.: The Stanford 3D scanning repository 5(10) (2005). https://graphics.stanford.edu/data/3Dscanrep/

  36. Li, L., Zhu, S., Fu, H., Tan, P., Tai, C.L.: End-to-end learning local multi-view descriptors for 3D point clouds (2020)

    Google Scholar 

  37. Lin, C.H., Kong, C., Lucey, S.: Learning efficient point cloud generation for dense 3D object reconstruction. In: AAAI Conference on Artificial Intelligence (AAAI) (2018)

    Google Scholar 

  38. Liu, S., Li, T., Chen, W., Li, H.: Soft rasterizer: a differentiable renderer for image-based 3D reasoning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019

    Google Scholar 

  39. Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., Sheikh, Y.: Neural volumes: learning dynamic renderable volumes from images. ACM Trans. Graph. 38(4), 65:1–65:14 (2019)

    Google Scholar 

  40. Loper, M.M., Black, M.J.: OpenDR: an approximate differentiable renderer. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 154–169. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10584-0_11

    Chapter  Google Scholar 

  41. Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3D surface construction algorithm. ACM SIGGRAPH Comput. Graph. 21(4), 163–169 (1987)

    Article  Google Scholar 

  42. Mahalanobis, P.C.: On the generalized distance in statistics. In: Proceedings of the National Institute of Sciences (Calcutta), pp. 49–55 (1936)

    Google Scholar 

  43. Martin, W.N., Aggarwal, J.K.: Volumetric descriptions of objects from multiple views. IEEE Trans. Patt. Anal. Mach. Intell. PAMI-5(2), 150–158 (1983). https://doi.org/10.1109/TPAMI.1983.4767367

  44. Max, N.: Optical models for direct volume rendering. IEEE Trans. Visual. Comput. Graph. 1(2), 99–108 (1995). https://doi.org/10.1109/2945.468400

    Article  Google Scholar 

  45. McGuire, M., Bavoil, L.: Weighted blended order-independent transparency. J. Comput. Graph. Tech. (JCGT) 2(2), 122–141 (2013). https://jcgt.org/published/0002/02/09/

  46. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NERF: representing scenes as neural radiance fields for view synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) Computer Vision - ECCV 2020, pp. 405–421. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_24

  47. Miller, I.D., et al.: Mine tunnel exploration using multiple quadrupedal robots (2020)

    Google Scholar 

  48. Muraki, S.: Volumetric shape description of range data using “blobby model”. In: Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques, pp. 227–235. SIGGRAPH 1991. Association for Computing Machinery, New York, NY, USA (1991). https://doi.org/10.1145/122718.122743

  49. Nimier-David, M., Vicini, D., Zeltner, T., Jakob, W.: Mitsuba 2: a retargetable forward and inverse renderer. ACM Trans. Graph. 38(6) (2019). https://doi.org/10.1145/3355089.3356498

  50. O’Meadhra, C., Tabib, W., Michael, N.: Variable resolution occupancy mapping using Gaussian mixture models. IEEE Robot. Autom. Lett. 4(2), 2015–2022 (2019). https://doi.org/10.1109/LRA.2018.2889348

    Article  Google Scholar 

  51. Pfister, H., Zwicker, M., van Baar, J., Gross, M.: Surfels: surface elements as rendering primitives. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 335–342. SIGGRAPH 2000. ACM Press/Addison-Wesley Publishing Co., USA (2000). https://doi.org/10.1145/344779.344936

  52. Ravi, N., et al.: Accelerating 3D deep learning with pytorch3d. arXiv:2007.08501 (2020)

  53. Reizenstein, J., Shapovalov, R., Henzler, P., Sbordone, L., Labatut, P., Novotny, D.: Common objects in 3D: large-scale learning and evaluation of real-life 3D category reconstruction. In: International Conference on Computer Vision (2021)

    Google Scholar 

  54. Rusinkiewicz, S., Levoy, M.: Efficient variants of the ICP algorithm. In: Proceedings Third International Conference on 3D Digital Imaging & Modeling, pp. 145–152 (2001). https://doi.org/10.1109/IM.2001.924423

  55. Schönberger, J.L., Zheng, E., Pollefeys, M., Frahm, J.M.: Pixelwise view selection for unstructured multi-view stereo. In: European Conference on Computer Vision (ECCV) (2016). https://doi.org/10.1007/978-3-319-46487-9_31

  56. Sculley, D.: Web-scale k-means clustering. In: Proceedings of the 19th International Conference on World Wide Web, pp. 1177–1178. WWW 2010. Association for Computing Machinery, New York, NY, USA (2010). https://doi.org/10.1145/1772690.1772862

  57. Shankar, K.S., Michael, N.: MRFMap: online probabilistic 3D mapping using forward ray sensor models. In: Robotics: Science and Systems (2020)

    Google Scholar 

  58. Sutherland, I.E., Sproull, R.F., Schumacker, R.A.: A characterization of ten hidden-surface algorithms. ACM Comput. Surv. 6(1), 1–55 (1974). https://doi.org/10.1145/356625.356626

  59. Szécsi, L., Illés, D.: Real-time metaball ray casting with fragment lists. In: Eurographics (2012)

    Google Scholar 

  60. Tabib, W., O’Meadhra, C., Michael, N.: On-manifold GMM registration. IEEE Robot. Autom. Lett. 3(4), 3805–3812 (2018). https://doi.org/10.1109/LRA.2018.2856279

    Article  Google Scholar 

  61. Teed, Z., Deng, J.: RAFT: recurrent all-pairs field transforms for optical flow. In: 16th European Conference on Computer Vision, pp. 402–419, Germany (2020). https://doi.org/10.1007/978-3-030-58536-5_24

  62. Teed, Z., Deng, J.: Droid-SLAM: deep visual SLAM for monocular, stereo, and RGB-D cameras (2021)

    Google Scholar 

  63. Tewari, A., et al.: MoFA: model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 3735–3744 (2017). https://doi.org/10.1109/ICCV.2017.401

  64. Tomic, T., et al.: Toward a fully autonomous UAV: research platform for indoor and outdoor urban search and rescue. IEEE Robot. Autom. Mag. 19(3), 46–56 (2012). https://doi.org/10.1109/MRA.2012.2206473

  65. Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment – a modern synthesis. In: Triggs, B., Zisserman, A., Szeliski, R. (eds.) Vision Algorithms: Theory and Practice, pp. 298–372. Springer, Berlin, Heidelberg (2000). https://doi.org/10.1007/3-540-44480-7_21

  66. Tsai, C., Sankaranarayanan, A., Gkioulekas, I.: Beyond volumetric albedo. In: CVPR, June 2019

    Google Scholar 

  67. Tucker, R., Snavely, N.: Single-view view synthesis with multiplane images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 551–560 (2020)

    Google Scholar 

  68. Wang, A., Wang, P., Sun, J., Kortylewski, A., Yuille, A.: VoGE: a differentiable volume renderer using gaussian ellipsoids for analysis-by-synthesis. arXiv preprint arXiv:2205.15401 (2022)

  69. Westman, E., Gkioulekas, I., Kaess, M.: Volumetric albedo framework for 3D imaging sonar. In: ICRA (2020)

    Google Scholar 

  70. Wyvill, G., McPheeters, C., Wyvill, B.: Data structure forsoft objects. Vis. Comput. 2(4), 227–234 (1986). https://doi.org/10.1007/BF01900346

    Article  Google Scholar 

  71. Wyvill, G., Trotman, A.: Ray-tracing soft objects. In: Chua, T.S., Kunii, T.L. (eds.) CG International, pp. 469–476. Springer Japan, Tokyo (1990). https://doi.org/10.1007/978-4-431-68123-6_27

  72. Yang, J., Li, H., Jia, Y.: Go-ICP: solving 3D registration efficiently and globally optimally. In: 2013 IEEE International Conference on Computer Vision, pp. 1457–1464 (2013). https://doi.org/10.1109/ICCV.2013.184

  73. Yang, S., Scherer, S.: CubeSLAM: Monocular 3-D object slam. IEEE Trans. Rob. 35(4), 925–938 (2019). https://doi.org/10.1109/TRO.2019.2909168

    Article  Google Scholar 

  74. Yifan, W., Serena, F., Wu, S., Öztireli, C., Sorkine-Hornung, O.: Differentiable surface splatting for point-based geometry processing. ACM Trans. Graph. 38(6), 1–14 (2019). https://doi.org/10.1145/3355089.3356513

    Article  Google Scholar 

  75. Zhang, C., Miller, B., Yan, K., Gkioulekas, I., Zhao, S.: Path-space differentiable rendering. ACM Trans. Graph. 39(4), 143:1–143:19 (2020). https://doi.org/10.1145/3386569.3392383

  76. Zhong, E.D., Lerer, A., Davis, J.H., Berger, B.: CryoDRGN2: Ab initio neural reconstruction of 3D protein structures from real cryo-EM images. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4066–4075, October 2021

    Google Scholar 

  77. Zhou, Q.Y., Park, J., Koltun, V.: Open3D: a modern library for 3D data processing. arXiv:1801.09847 (2018)

  78. Zhou, Q., Jacobson, A.: Thingi10k: a dataset of 10, 000 3D-printing models. CoRR abs/1605.04797 (2016). arxiv:1605.04797

  79. Zwicker, M., Pfister, H., van Baar, J., Gross, M.: Surface splatting. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 371–378. SIGGRAPH 2001 (2001). https://doi.org/10.1145/383259.383300

Download references

Author information

Authors and Affiliations

  1. Carnegie Mellon University, Pittsburgh, PA, USA

    Leonid Keselman & Martial Hebert

Authors
  1. Leonid Keselman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Martial Hebert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Leonid Keselman .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1663 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Keselman, L., Hebert, M. (2022). Approximate Differentiable Rendering with Algebraic Surfaces. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13692. Springer, Cham. https://doi.org/10.1007/978-3-031-19824-3_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19824-3_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19823-6

  • Online ISBN: 978-3-031-19824-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation