Skip to main content
SpringerLink
Log in

A Multi-Image Shape-from-Shading Framework for Near-Lighting Perspective Endoscopes

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

This article formulates a near-lighting shape-from-shading problem with a pinhole camera (perspective projection) and presents a solution to reconstruct the Lambertian surface of bones using a sequence of overlapped endoscopic images, with partial boundaries in each image. First we extend the shape-from-shading problem to deal with perspective projection and near point light sources that are not co-located with the camera center. Secondly we propose a multi-image framework which can align partial shapes obtained from different images in the world coordinates by tracking the endoscope. An iterative closest point (ICP) algorithm is used to improve the matching and recover complete occluding boundaries of the bone. Finally, a complete and consistent shape is obtained by simultaneously re-growing the surface normals and depths in all views. In order to fulfill our shape-from-shading algorithm, we also calibrate both geometry and photometry for an oblique-viewing endoscope that are not well addressed before in the previous literatures. We demonstrate the accuracy of our technique using simulations and experiments with artificial bones.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Besl, P. J., & McKay, N. D. (1992). A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2), 239–256.

    Article  Google Scholar 

  • Clarkson, M. J., Rueckert, D., King, A. P., Edwards, P. J., Hill, D. L. G., & Hawkes, D. J. (1999). Registration of video images to tomographic images by optimising mutual information using texture mapping. In LNCS: Vol. 1679. Proceedings of the second international conference on medical image computing and computer-assisted intervention (MICCAI’99) (pp. 579–588). Berlin: Springer.

    Chapter  Google Scholar 

  • Courteille, F., Crouzil, A., Durou, J.-D., & Gurdjos, P. (2004). Towards shape from shading under realistic photographic conditions. Proceedings of the 17th International Conference on Pattern Recognition (ICPR’04), 2, 277–280.

    Article  Google Scholar 

  • Dey, D., Gobbi, D. G., Slomka, P. J., Surry, K. J. M., & Peters, T. M. (2002). Automatic fusion of freehand endoscopic brain images to three-dimensional surfaces: Creating stereoscopic panoramas. IEEE Transactions on Medical Imaging, 21(1), 23–30.

    Article  Google Scholar 

  • Durou, J.-D., Falcone, M., & Sagona, M. (2008). Numerical methods for shape-from-shading: A new survey with benchmarks. Computer Vision and Image Understanding, 109(1), 22–43.

    Article  Google Scholar 

  • Forster, C. H. Q., & Tozzi, C. L. (2000). Toward 3d reconstruction of endoscope images using shape from shading. In Proceedings of the 13th Brazilian symposium on computer graphics and image processing (SIBGRAPHI’00) (pp. 90–96).

  • Frankot, R. T., & Chellappa, R. (1988). A method for enforcing integrability in shape from shading algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 10(4), 439–451.

    Article  MATH  Google Scholar 

  • Fuchs, H., Livingston, M. A., Raskar, R., Colucci, D., Keller, K., State, A., Crawford, J. R., Rademacher, P., Drake, S. H., & Meyer, A. A. (1998). Augmented reality visualization for laparoscopic surgery. In LNCS: Vol. 1496. Proceedings of the first international conference on medical image computing and computer-assisted intervention (MICCAI’98) (pp. 934–943). Berlin: Springer.

    Google Scholar 

  • Hartley, R., & Zisserman, A. (2004). Multiple view geometry in computer vision (2nd edn.). Cambridge: Cambridge University Press.

    MATH  Google Scholar 

  • Hasegawa, J. K., & Tozzi, C. L. (1996). Shape from shading with perspective projection and camera calibration. Computers and Graphics, 20(3), 351–364.

    Article  Google Scholar 

  • Horn, B. K. P., & Brooks, M. J. (1986). The variational approach to shape from shading. Computer Vision, Graphics, and Image Processing (CVGIP’86), 33(2), 174–208.

    Article  Google Scholar 

  • Horn, B. K. P., & Brooks, M. J. (1989). Shape from shading. Cambridge: MIT.

    Google Scholar 

  • Horn, B. K. P., & Sjoberg, R. W. (1979). Calculating the reflectance map. Applied Optics, 18(11), 1770–1779.

    Article  Google Scholar 

  • Ikeuchi, K., & Horn, B. K. P. (1981). Numerical shape from shading and occluding boundaries. Artificial Intelligence, 17(1-3), 141–184.

    Article  Google Scholar 

  • Iyengar, A. K. S., Sugimoto, H., Smith, D. B., & Sacks, M. S. (2001). Dynamic in vitro quantification of bioprosthetic heart valve leaflet motion using structured light projection. Annals of Biomedical Engineering, 29(11), 963–973.

    Article  Google Scholar 

  • Kimmel, R., & Sethian, J. A. (2001). Optimal algorithm for shape from shading and path planning. Journal of Mathematical Imaging and Vision, 14(3), 237–244.

    Article  MATH  MathSciNet  Google Scholar 

  • Kozera, R. (1991). Existence and uniqueness in photometric stereo. Applied Maths and Computation, 44(1), 1–103.

    Article  MATH  MathSciNet  Google Scholar 

  • Kozera, R. (1992). On shape recovery from two shading patterns. International Journal of Pattern Recognition and Artificial Intelligence, 6(4), 673–698.

    Article  Google Scholar 

  • Kozera, R. (1998). An overview of the shape from shading problem. Machine Graphics and Vision, 7(1), 291–312.

    MathSciNet  Google Scholar 

  • Kozera, R., & Noakes, L. (2004). Noise reduction in photometric stereo with non-distant light sources. Proceedings of the International Conference on Computer Vision and Graphics (ICCVG’04), 32, 103–110.

    Google Scholar 

  • Leclerc, Y. G., & Bobick, A. F. (1991). The direct computation of height from shading. In Proceedings of IEEE computer society conference on computer vision and pattern recognition (CVPR’91) (pp. 552–558).

  • Lee, K. M., & Kuo, C.-C. J. (1994). Shape from shading with perspective projection. Computer Vision, Graphics, and Image Processing: Image Understanding, 59(2), 202–212.

    Article  Google Scholar 

  • Lee, K. M., & Kuo, C.-C. J. (1997). Shape from shading with a generalized reflectance map model. Computer Vision and Image Understanding, 67(2), 143–160.

    Article  Google Scholar 

  • Litvinov, A., & Schechner, Y. Y. (2005). Addressing radiometric nonidealities: a unified framework. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 2, 52–59.

    Article  Google Scholar 

  • Mourgues, F., Devernay, F., Malandain, G., & Coste-Manière, E. (2001). 3d reconstruction of the operating field for image overlay in 3d-endoscopic surgery. In: Proceedings of the IEEE and ACM international symposium on augmented reality (ISAR’01) (pp. 191–192).

  • Okatani, T., & Deguchi, K. (1997). Shape reconstruction from an endoscope image by shape from shading technique for a point light source at the projection center. Computer Vision and Image Understanding, 66(2), 119–131.

    Article  Google Scholar 

  • Penna, M. A. (1989). A shape from shading analysis for a single perspective image of a polyhedron. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(6), 545–554.

    Article  Google Scholar 

  • Poelman, C. J., & Kanade, T. (1997). A paraperspective factorization method for shape and motion recovery. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(3), 206–218.

    Article  Google Scholar 

  • Pollefeys, M., Koch, R., & Gool, L. V. (1999). Self-calibration and metric reconstruction inspite of varying and unknown intrinsic camera parameters. International Journal of Computer Vision, 32(1), 7–25.

    Article  Google Scholar 

  • Prados, E., & Faugeras, O. (2003). Perspective shape from shading and viscosity solution. Proceedings of the Ninth IEEE International Conference on Computer Vision (ICCV’03), 2, 826–831.

    Article  Google Scholar 

  • Prados, E., & Faugeras, O. (2005). Shape from shading: a well-posed problem? Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 2, 870–877.

    Google Scholar 

  • Samaras, D., & Metaxas, D. N. (1999). Coupled lighting direction and shape estimation from single images. Proceedings of the International Conference on Computer Vision (ICCV’99), 2, 868–874.

    Article  Google Scholar 

  • Seshamani, S., Lau, W., & Hager, G. (2006). Real-time endoscopic mosaicking. In LNCS: Vol. 4190. Proceedings of the Ninth International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI’06) (pp. 355–363). Berlin: Springer.

    Google Scholar 

  • Stoyanov, D., Darzi, A., & Yang, G. Z. (2005). A practical approach towards accurate dense 3d depth recovery for robotic laparoscopic surgery. Computer Aided Surgery, 10(4), 199–208.

    Article  Google Scholar 

  • Tankus, A., Sochen, N., & Yeshurun, Y. (2003). A new perspective [on] shape-from-shading. Proceedings of the Ninth IEEE International Conference on Computer Vision (ICCV’03), 2, 862–869.

    Article  Google Scholar 

  • Tankus, A., Sochen, N., & Yeshurun, Y. (2004). Reconstruction of medical images by perspective shape-from-shading. Proceedings of the Pattern Recognition, 17th International Conference on (ICPR’04), 3, 778–781.

    Article  Google Scholar 

  • Tankus, A., Sochen, N., & Yeshurun, Y. (2005). Shape-from-shading under perspective projection. International Journal of Computer Vision, 63(1), 21–43.

    Article  Google Scholar 

  • Tsai, R. Y. (1987). A versatile camera calibration technique for high-accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses. IEEE Journal of Robotics and Automation, 3, 323–344.

    Article  Google Scholar 

  • Woodham, R. J. (1980). Photometric method for determining surface orientation from multiple images. Optical Engineering, 19(1), 139–144.

    Google Scholar 

  • Yamaguchi, T., Nakamoto, M., Sato, Y., Konishi, K., Hashizume, M., Sugano, N., Yoshikawa, H., & Tamura, S. (2004). Development of a camera model and calibration procedure for oblique-viewing endoscopes. Computer Aided Surgery, 9(5), 203–214.

    Article  Google Scholar 

  • Zhang, R., Tsai, P. S., Cryer, J. E., & Shah, M. (1999). Shape from shading: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(8), 690–706.

    Article  Google Scholar 

  • Zhang, Z. (1998). A flexible new technique for camera calibration (Technical Report MSR-TR-98-71). Microsoft Research.

Download references

Author information

Authors and Affiliations

  1. Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA, 15213, USA

    Chenyu Wu, Srinivasa G. Narasimhan & Branislav Jaramaz

  2. Institute of Computer Assisted Surgery, 3380 Boulevard of the Allies, Suite 270, Pittsburgh, PA, 15213, USA

    Branislav Jaramaz

Authors
  1. Chenyu Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Srinivasa G. Narasimhan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Branislav Jaramaz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chenyu Wu.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wu, C., Narasimhan, S.G. & Jaramaz, B. A Multi-Image Shape-from-Shading Framework for Near-Lighting Perspective Endoscopes. Int J Comput Vis 86, 211–228 (2010). https://doi.org/10.1007/s11263-009-0207-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-009-0207-3

Keywords

Search

Navigation