Skip to main content
SpringerLink
Log in

Toward Human-centered XAI in Practice: A survey

  • Review
  • Published:
Machine Intelligence Research Aims and scope Submit manuscript

Abstract

Human adoption of artificial intelligence (AI) technique is largely hampered because of the increasing complexity and opacity of AI development. Explainable AI (XAI) techniques with various methods and tools have been developed to bridge this gap between high-performance black-box AI models and human understanding. However, the current adoption of XAI technique still lacks “human-centered” guidance for designing proper solutions to meet different stakeholders’ needs in XAI practice. We first summarize a human-centered demand framework to categorize different stakeholders into five key roles with specific demands by reviewing existing research and then extract six commonly used human-centered XAI evaluation measures which are helpful for validating the effect of XAI. In addition, a taxonomy of XAI methods is developed for visual computing with analysis of method properties. Holding clearer human demands and XAI methods in mind, we take a medical image diagnosis scenario as an example to present an overview of how extant XAI approaches for visual computing fulfil stakeholders’ human-centered demands in practice. And we check the availability of open-source XAI tools for stakeholders’ use. This survey provides further guidance for matching diverse human demands with appropriate XAI methods or tools in specific applications with a summary of main challenges and future work toward human-centered XAI in practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. A. Maier, C. Syben, T. Lasser, C. Riess. A gentle introduction to deep learning in medical image processing. Zeitschrift fur Medizinische Physik, vol. 29, no. 2, pp. 86–101, 2019. DOI: https://doi.org/10.1016/j.zemedi.2018.12.003.

    Article  Google Scholar 

  2. E. Gibson, W. Q. Li, C. Sudre, L. Fidon, D. I. Shakir, G. T. Wang, Z. Eaton-Rosen, R. Gray, T. Doel, Y. P. Hu, T. Whyntie, P. Nachev, M. Modat, ID. C. Barratt, S. Ourselin, M. J. Cardoso, T. Vercauteren. Nifty Net: A deep-learning platform for medical imaging. Computer Methods and Programs in Biomedicine, vol. 158, pp. 113–122, 2018. DOI: https://doi.org/10.1016/j.cmpb.2018.01.025.

    Article  Google Scholar 

  3. A. M. Ozbayoglu, M. U. Gudelek, O. B. Sezer. Deep learning for financial applications: A survey. Applied Soft Computing, vol 93, Article number 106384, 2020. DOI: https://doi.org/10.1016/j.asoc.2020.106384.

  4. J. H. Zhang, J. R. Zhai, H. B. Wang. A survey on deep learning in financial markets. In Proceedings of the 1st International Forum on Financial Mathematics and Financial Technology, Springer, Smgapoet, pp. 35–57, 2021. DOI: https://doi.org/10.1007/978-981-15-8373-5_3.

    Google Scholar 

  5. P. Prajwal, D. Prajwal, D. H. Harish, R. Gajanana, B. S. Jayasri, S. Lokesh. Object detection in self driving cars using deep learning. In Proceedings of International Conference on Innovative Computing, Intelligent Communication and Smart Electrical Systems, IEEE, Chennai, India, pp. 1–7, 0221. OOI: 10.1199/ICES2305.2021.9633965.

    Google Scholar 

  6. Q. Rao, J. Frtunikj. Deep learnmg for self-driving cars: Chances and chaHenges. In Proceedings of IEEE/ACM the 1st International Workshop on Software Engineering for AI in Autonomous Systems, IEEE, Goathenburg, Sweden, pp. 35–38, 2018.

    Google Scholar 

  7. E. Fersini, F. Archetti, E. Messina. Towards a smooth e-justice: Semantic models and machine learning. Integration of Practice-oriented Knowledge Technology: Trends and Prospectives, Fathi, Ed., Berlin, Germany: Springer, pp. 57–70, 2013. DOI: https://doi.org/10.1007/978-3-642-34471-8_5.

    Chapter  Google Scholar 

  8. Y. J. Wang, J. Gao, J. J. Chen. Deep learning algorithm for judicial judgment prediction based on BERT. In Proceedings of the 5th International Conference on Computing, Communication and Security, IEEE, Patna, India, 2020. DOI: https://doi.org/10.1109/ICCCS49678.2020.9277068.

    Book  Google Scholar 

  9. B. G. Chen, Y. Li, S. Zhang, H. Lian, T. K He. A deep learning method for judicial decision support. In Proceedings of the 19th IEEE International Conference on Software Quality, Reliability and Security Companion, Sofia, Bulgaria, pp. 145–149, 2019. DOI: https://doi.org/10.1109/QRS-C.2019.00040.

  10. C. Manresa-Yee, M. F. Roig-Maimó, S. Ramis, R. Mas-Sansó. Advances in XAI: Explanation interfaces in healthcare. Handbook of Artificial Intelligence in Healthcare: Vol 2: Practicalities and Prospects, C. P. Lim, Y. W. Chen, A. Vaidya, C. Mahorkar, L. C. Jain, Eds., Cham, Germany: Springer, pp. 357–369, 2022. DOI: https://doi.org/10.1007/978-3-030-83620-7_15.

    Chapter  Google Scholar 

  11. D. Gunning, E. Vorm, J. Y. Wang, M. Turek. DARPA’s explainable AI (XAI) program: A retrospective. Applied AI Letters, vol. 2, no. 4, Article number e61, 2021. DOI: https://doi.org/10.1002/ail2.61.

  12. T. MiHer. Explanation in artfficial intelligence: Insights from the sodal sdences. Artificial Intelligence, vol. 267, pp. 1–38, 2019. DOI: https://doi.org/10.1016/j.artint.2018.07.007.

    Article  MathSciNet  Google Scholar 

  13. P. Voigt, A. Von Dem Bussche. The EU General Data Protection Regulation (GDPR): A Practical Guide, Cham, Germany: Springer, 2017. DOI: https://doi.org/10.1007/978-3-319-57959-7.

    Book  Google Scholar 

  14. R. R. Selvaraju, M. Cogswell, A. Das, R Vedantam, D. Parikh, D. Batra. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In Proceedings of IEEE International Conference on Computer Vision, Venice, Italy, pp. 618–626, 2017. DOI: https://doi.org/10.1109/ICCV.2017.74.

  15. X. W. Kong, X. Z. Tang, Z. M. Wang. A survey of explainable artificial intelligence decision Systems Engineering — Theory & Practice, vol. 41, no. 2, pp. 524–536, 2021. DOI: https://doi.org/10.12011/SETP2020-1536. (in Chinese)

    Google Scholar 

  16. C. Liu, X. W. Sun, J. D. Wang, H. Y. Tang, T. Li, T. Qin, W. Chen, T. Y. Liu. Learning causal semantic representation for out-of-distribution prediction. In Proceedings of the 35th Annual Conference on Neural Information Processing Systems, Virtual, pp. 6155–6170, 2021.

  17. B. L. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba. Learning deep features for discriminative localization. In Proceedings of IEEE Conference on Computer Vision and Pattern Recogn t on, Las Vegas, USA, pp. 2921–2929, 2016. DOI: https://doi.org/10.1109/CVPR.2016.319.

  18. M. Alber, S. Lapuschkin, P. Seegerer, M. Hägele, K. T. Schutt, G. Montavon, W. Samek, K. R. Müller, S. Dähne, P. J. Kindermans. iNNvestigate neural networks!. Journal of Machine Learning Research, vol. 20, no.93, pp. 1–8, 2019.

    MathSciNet  Google Scholar 

  19. H. Baniecki, W. Kretowicz, P. Piątyszek, J. Wisniewski, P. Biecek. Dalex: Responsible machine learning with interactive explainability and fairness in python. The Journal of Machine Learning Research, vol. 22, no. 214, pp. 1–7, 2021.

    MathSciNet  Google Scholar 

  20. D. V. Carvalho, E. M. Pereira, J. S. Cardoso. Machine learning interpretability: A survey on methods and metrics. Electronics, vol. 8, no. 8, Article number 832, 2019. DOI: https://doi.org/10.3390/electronics8080832.

  21. A. Singh, S. Sengupta, V. Lakshminarayanan. Explainable deep learning models in medical image analysis. Journal of Imaging, vol. 6, no. 6, Article number 52, 2020. DOI: https://doi.org/10.3390/jimaging6060052.

  22. Y. Zhang, P. Tiňo, A. Leonardis, K. Tang. A survey on neural network interpretability. IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 5, no. 5, pp. 726–742, 2021. DOI: https://doi.org/10.1109/TETCI.2021.3100641.

    Article  Google Scholar 

  23. U. Bhatt, A. Xiang, S. Sharma, A. Weller, A. Taly, Y. H. Jia, J. Ghosh, R. Puri, J. M. F. Moura, P. Eckersley. Explainable machine learning in deployment. In Proceedings of Conference on Fairness, Accountability, and Transparency ACM Barcelona Spain pp. 648–657 2020. DOI: https://doi.org/10.1145/3351095.3375624.

  24. U. Ehsan, M. O. Riedl. Human-centered explainable AI: Towards a reflective sociotechnical approach. In Proceedings of the 22nd International Conference on Human-Computer Interaction, Springer, Copenhagen, Denmark, pp. 449–466, 2020. DOI: https://doi.org/10.1007/978-3-030-60117-1_33.

    Google Scholar 

  25. T. A. J. Schoonderwoerd, W. Jorritsma, M. A. Neerincx, K. Van Den Bosch. Human-centered XAI: Developing design patterns for explanations of clinical decision support systems. International Journal of Human-computer Studies, vol. 154, Article number 102684, 2021. DOI: https://doi.org/10.1016/j.ijhcs.2021.102684.

  26. A. Alshehri, T. Miller, M. Vered, H. Alamri. Human centered explanation for goal recognition system. In Proceedings of IJCAI-PRICAI Workshop on Explainable Artificial Intelligence, Japan, Article number 7, 2021.

  27. J. C. Zhu, A. Liapis, S. Risi, R. Bidarra, G. M. Young-blood. Explainable AI for desggners: A human-centered perspective on mixed-initlative co-crearion. In Proceedings of IEEE Conference on Computational Intelligence and Games, Maastricht, Netherlands, pp. 1–8, 2018. DOI: https://doi.org/10.1109/CIG.2018.8490433.

    Google Scholar 

  28. B. Babic, S. Gerke, T. Evgeniou, I. G. Cohen. Beware explanations room AI in health care. Science, vol. 373, no. 6552, pp. 284–286, 2021 DOI: https://doi.org/10.1126/science.abg1834.

    Article  Google Scholar 

  29. T. Vermeire, T. Laugel, X. Renard, D. Martens, M. Detyniecki. How to choose an explainability method? Towards a methodical implementation of XAI in practice. In Proceedings of Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, pp.521-533, 2021 DOI: 10.1007/978-3-030-93736-2_39.

  30. M. Langer, D. Oster, T. Speith, H. Hermanns, L. Kästner, E. Schmidt, A. Sesing, K. Baum. What do we want from explainable artificial intelligence (XAI)? — A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research. Artificial Intelligence, vol. 296, Article number 103473, 2021. DOI: https://doi.org/10.1016/j.artint.2021.103473.

  31. M. R. Islam, M. U. Ahmed, S. Barua, S. Begum. A systematic review of explainable artificial intelligence in terms of different application domains and tasks. Applied Sciences, vol. 12, no. 3, Article number 1353, 2022. DOI: https://doi.org/10.3390/app12031353.

  32. C. Bove, J. Aigrain, M. J. Lesot, C. Tijus, M. Detyniecki. Contextualization and exploration of local feature importance explanations to improve understanding and satisfaction of non-expert users. In Proceedings of the 27th International Conference on Intelligent User Interfaces, ACM, Helsinki, Finland, pp. 807–819, 2022. DOI: https://doi.org/10.1145/3490099.3511139.

    Google Scholar 

  33. S. Dey, P. Chakraborty, B. C. Kwon, A. Dhurandhar, M. Ghalwash, F. J. S. Saiz, K. Ng, D. Sow, K. R. Varshney, P. Meyer. Human-centered explainability for life sciences, healthcare, and medical informatics. Patterns, vol. 3, no. 5, Article number 100493, 2022. DOI: 1011016/j.patter.2022.100493.

  34. R Tomsett, D Braines, D Harborne, A D Preece, S Chakraborty. Interpretable to whom? A role-based model for analyzing interpretable machine learning systems In Proceedings of the 3rd Annual Workshop on Human Inteapaetability in Machine Learning, Stockholm, Sweden, 2018.

  35. J. Posada, C. Toro, I. Barandiaran, D. Oyarzun, Stricker, R. De Amicis, E. B. Pinto, P. Eisert, J. Döllner, I. Vallarino. Visual computing as a key enabling technology for industrie 4.0 and industrial internet. IEEE Computer Graphics and Applications, vol. 35, no. 2, pp. 26–40, 2015. DOI: https://doi.org/10.1109/MCG.2015.45.

    Article  Google Scholar 

  36. K. Simonyan, A. Vedaldi, A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Proceedings of the 2nd International Conference on Learning Representations, Banff, Canada, 2014. DOI: arxiv.org/abs/1312.6034.

  37. B. Kim, M. Wattenberg, J. Gilmer, C. J. Cai, J. Wexler, F. Viégas, R. Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, pp. 2673–2682, 2018.

  38. M. Chromik, A. Butz. Human-XAI interaction: A review and design principles for explanation user interfaces. In Proceedings of the 18th IFIP Conference on Human-Computer Interaction, Springer, Bari, Italy, pp. 619–640, 2021. DOI: https://doi.org/10.1007/978-3-030-85616-8_36.

    Google Scholar 

  39. A. Adadi, M. Berrada. Peeking inside the Mack-box: A survey on explainable artificial intelligence (XAI). IEEE Access, vol. 6, pp. 52138–52160, 2018. DOI: https://doi.org/10.1109/ACCESS.2018.2870052.

    Article  Google Scholar 

  40. R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, D. Pedreschi. A survey of methods for explaining black box modete. ACM Computing Surveys, vol. 51, no. 5, Article number 93, 2019. DOI: https://doi.org/10.1145/3236009.

  41. A. B. Arrieta, N. Daaz-Rodriguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. Garcia, S. Gil-Lopez, D. Molina, R. Benjamins, R. Chatila, F. Herrera. Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, vol. 58, pp. 82–115, 2020. DOI: https://doi.org/10.1016/j.inffus.2019.12.012.

    Article  Google Scholar 

  42. C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, vol. 1, no. 5, pp. 206–215, 2019. DOI: https://doi.org/10.1038/s42256-019-0048-x.

    Article  Google Scholar 

  43. Y. F. Zhang, X. Chen. Explainable recommendation: A survey and new perspectives. Foundations and Trends in Information Retrieval, vol. 14, no. 1, pp. 1–101, 2020. DOI: https://doi.org/10.1561/1500000066.

    Article  Google Scholar 

  44. A. Holzinger, G. Langs, H. Denk, K. Zatloukal, H. Müller. Causability and explainability of artificial intelligence in medicine. WIREs Data Mining and Knowledge Discovery, vol. 9, no. 4, Article number e1312, 2019. DOI: https://doi.org/10.1002/widm.1312.

  45. C. Meske, E. Bunde, J. Schneider, M. Gersch. Explainable artificial intelligence: Objectives, stakeholders, and future research opportunities. Information Systems Management, vol. 39, no. 1, pp. 53–63, 2022. DOI: https://doi.org/10.1080/10580530.2020.1849465.

    Article  Google Scholar 

  46. I. Siddavatam, A. Dalvi, V. Thakkar, A. Vedpathak, S. Moradiya, A. Jain. Explainability using decision trees and monte carlo simulations. In Proceedings of the 4th International Conference on Advances in Science & Technology, Mumbai, India, 2021. DOI: https://doi.org/10.2139/ssrn.3868707.

  47. S. Mohseni, N. Zarei, E. D. Ragan. A multidisciplinary survey and framework for design and evaluation of explainable ai systems. ACM Transactions on Interactive Intelligent Systems, vol. 11, no. 3–4, Article number 24, 2021. DOI: https://doi.org/10.1145/3387166.

  48. H. Suresh, S. R. Gomez, K. K. Nam, A. Satyanarayan. Beyond expertise and roles: A framework to characterize the stakeholders of interpretable machine learning and their needs. In Proceedings of CHI Conference on Human Factors in Computing Systems, ACM, Yokohama, Japan, Article number 74, 2021. DOI: https://doi.org/10.1145/3411764.3445088.

    Google Scholar 

  49. U. Ehsan, P. Wintersberger, Q. V. LIuc,, M. Mara, M. I. Streit, S. Wachter, A. Riener, M. O. Riedl. Operationalizing human-centered perspect ves n expla nable AI. In Proceedings of Extended Abstracts of CHI Conference on Human Factors in Computing Systems, ACM, Yokohama, Japan, Artide number 94, 2021. DOI: https://doi.org/10.1145/3411763.3441342.

    Google Scholar 

  50. C. T. Wolf. Explainability scenarios: Towards scenario-based XAI design. In Proceedings of the 24th International Conference on Intelligent User Interfaces, ACM, Marina del Ray, USA, pp.S22–S5V, 2019. DOI https://doi.org/10.1145/3301275.3302317.

    Google Scholar 

  51. F. Hohman, M. S. Kahng, R. Pienta, D. HS Chau. Visual analyt cs n deep learn ng: An nterrogat ve survey for the next frontiers. IEEE Transactions on Visualization and Computer Graphics, vol. 25, no. 8, pp. 2674–2693, 2049. DOI: https://doi.org/10.1109/TVCG.2018.2843369.

    Article  Google Scholar 

  52. R. L. Yu, L. Shi. A user-based taxonomy for deep learning visualization. Visual Informatics, vol. 2, no. 3, pp. 147–154, 2018. DOI: https://doi.org/10.1016/j.visinf.2018.09.001.

    Article  Google Scholar 

  53. L. X. Yang, H. R. Wang, L. A. Deteris. What does t mean to explain? A user-centered study on AI explainability. In Proceedings of the 2nd International Conference on Human-Computer Interaction, Springer, pp. 107–121, 2021. DOI: https://doi.org/10.1007/978-3-030-77772-28.

  54. S. Schulze-Weddige, TS Zylowski. User study on the effects expiainable AI vfeuaiiaations on non-exprttr. In Proceedings of International Conference on ArtsIT, Interactivity and Game Creation, Springer, pp. 457–467, 2022. DOI: https://doi.org/10.1007/978-3-030-95531-1_31.

  55. S. R. Hong, J. Hullman, E. Bertini. Human factors in model interpretability: indstry practices, challenges, and needs. Proceedings of the ACM on Human-Computer Interaction, vol. 4, no. CSCW1, Article number 68, 2020. DOI: https://doi.org/10.1145/3392878.

  56. M. Hind. Explaining explainable AI. XRDS: Crossroads, the ACM Magazine for Students, vol. 25, no. 3, pp. 16–19, 2019. DOI: https://doi.org/10.1145/3313096.

    Article  Google Scholar 

  57. S. Dhanorkar, C. T. Wolf, K. Qian, A. B. Xu, L. Popa, Y. Y. Li. Who needs to know what, when? Broadening the explainable AI (XAI) design space by looking at explanations across the AI lifecycle. In Proceedings of Designing Interactive Systems Conference, ACM, pp. 1591–1602, 2021. DOI: https://doi.org/10.1145/3461778.3462131.

  58. Q. V. Liao, D. Gruen, S. Miller. Questioning the AI: Informing design practices for explainable AI user experiences. in Proceedings of CHI Conference on Human Factors in Computing Systems, ACM, Honolulu, USA, Article number 463, 2020. DOI: https://doi.org/10.1145/3313831.3376590.

    Google Scholar 

  59. F. Doshi-Velez, B. Kim. Considerations for evaluation and generalization in interpretable machine learning. Explainable and Interpretable Models in Computer Vision and Machine Learning, H. J. Escalante, S. Escalera, I. Guyon, X. Baró, Y. Güçlütürk, U. Güçlü, M. Van Gerven, Eds., Cham, Germany: Springer, pp. 3–17, 2018. DOI: https://doi.org/10.1007/978-3-319-98131-4_1.

    Chapter  Google Scholar 

  60. M Chromik, M Schuessler A taxonomy for human subject evaluation of black-box explanations in XAI. In Proceedings of the Workshop on Explainable Smart Systems for Algorithmic Transparency in Emerging Technologies Co-located with 25th International Conference on Intelligent User Interfaces, Cagliari, Italy, Article number 7, 2020.

  61. A. F. Markus, J. A. Kors, P. R. Rijnbeek. The role of explainability in creating trustworthy artificial intelligence for health care: A comprehensive survey of the terminology, design choices, and evaluation strategies. Journal of Biomedical Informatics, vol. 113, Article number 103655, 2021. DOI: https://doi.org/10.1016/j.jbi.2020.103655.

  62. Y. Xie, M. Chen, D. Kao, G. Gao, X. A. Chen. CheX-plain: Enabling physicians to explore and understand data-driven, AI-enabled medical imaging analysis. in Proceedings of CHI Conference on Human Factors in Computing Systems, ACM, Honolulu, USA, Article number 678, 2020. DOI: https://doi.org/10.1145/3313831.3376807.

    Google Scholar 

  63. S. Tan, R. Caruana, G. Hooker, Y. Lou. Distill-and-compare: Auditing black-box models using transparent model distillation. In Proceedings of AAAI/ACM Conference on AI, Ethics, and Society, ACM, New Orleans, USA, pp. 303–310, 2018. DOI: https://doi.org/10.1145/3278721.3278725.

    Google Scholar 

  64. P. J. Sun, L. Wu, K. Zhang, Y. J. Fu, R. C. Hong, M. Wang. Dual teaming for explainable recommendation: Towards umfying user preference prediction and review generation. In Proceedings of the Web Conference, ACM, Taipei, China, pp. 837–847, 2020. DOI: https://doi.org/10.1145/3366423.3380164.

    Google Scholar 

  65. Q. Liu, S. Wu, L. Wang. DeepStyle: Learning user preferences for visual recommendation. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, Tokyo, Japan, pp.841–844, 2017. DOI: https://doi.org/10.1145/3077136.3080658.

    Google Scholar 

  66. J. D. Fuhrman, N. Gorre, Q. Y. Hu, H. Li, I. El Naqa, M. L. Giger. A review of explainable and interpretable AI with applications in COVID-19 imaging. Medical Physics, vol. 49, no. 1, pp. 1–14, 2022. DOI: https://doi.org/10.1002/mp.15359.

    Article  Google Scholar 

  67. J. Kim, A. Rohrbach, T. Darrell, J. Canny, Z. Akata. Textual explanations for self-driving vehicles. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 577–593, 2018. DOI: https://doi.org/10.1007/978-3-030-01216-835.

    Google Scholar 

  68. A. Deeks. The judicial demand for explainable artificial intelligence. Columbia Law Review, vol. 119, no. 7, pp. 1829–1850, 2019.

    Google Scholar 

  69. U. Ehsan, Q. V. Liao, M. Muller, M. O. Riedl, J. D. Weisz. Expanding explainability: Towards social transparency in AI systems. In Proceedings of CHI Conference on Human Factors in Computing Systems, ACM, Yokohama, Japan, Article number 82, 2021. DOI: https://doi.org/10.1145/3411764.34451883.

    Google Scholar 

  70. B. Goodman, S. Flaxman. European union regulations on algorithmic decision-making and a “right to explanation”. Al Magazine, vol. 38, no. 3, pp. 50–57, 2017. DOI: https://doi.org/10.1609/aimag.v38i3.2741.

    Article  Google Scholar 

  71. D. D. Wang, Q. Yang, A. Abdul, B. Y. Lim. Designing theory-driven user-centric explainable AI. In Proceedings of CHI Conference on Human Factors in Computing Systems, ACM, Glasgow, UK, Article number 601, 2019. DOI: https://doi.org/10.1145/3290605.3300831.

    Book  Google Scholar 

  72. S. T. Mueller, E. S. Veinott, R. R. Hoffman, G. Klein, L. Alam, T. Mamun, W. J. Clancey. Principles of explanation in human-AI systems. Online], Available: https://arxiv.org/abs/2102.04972, 2021.

  73. M. Franklin, D. Lagnado. Human—AI interadinn paradigm for evaluating explainable artificial intelligence. In Proceedings of the 24th International Conference on Human-computer Interaction, Springer, pp. 404–411, 2022. DOI: https://doi.org/10.1007/978-3-031-06417-354.

  74. G. Bansal, B. Nushi, E. Kamar, W. S. Lasecki, D. S. Weld, E. Horvitz. Beyond accuracy: The role of mental models in human — AI team performance. In Proceedings of the 7th AAAI Conference on Human Computation and Crowdsourcing, Stevenson, USA, pp. 2–11, 2019. DOI: https://doi.org/10.1609/hcomp.v7i1.5285.

  75. S. J. Maarten, L. G. Militello, T. Ormerod, L. Raanan. Macrocognition, mental models, and cognitive task analysis methodology. Naturalistic Decision Making and Macrocognition, L. Militello, R. Lipshitz, J. M. Schraagen, Eds., London, UK: CRC Press, pp. 57–80, 2008.

    Google Scholar 

  76. M. Chromik, M. Eiband, F. Buchner, A. Krüger, A. Butz. I think I get your point, AI! The illusion of explanatory depth in explainable AI. In Proceedings of the 26th International Conference on Intelligent User Interfaces, ACM, New York, USA, pp. 307–317, 2021. DOI: https://doi.org/10.1145/3397481.3450644.

    Google Scholar 

  77. H. F. Cheng, R. T. Wang, Z. Zhang, F. O’Connell, T. Gray, F. M. Harper, H. Y. Zhu. Explaining decision-making algorithms through UI: Strategies to help non-expert stakeholders. In Proceedings of CHI Conference on Human Factors in Computing Systems, ACM, Glasgow Scotland, UK, Article number 559, 2019. DOI: https://doi.org/10.1145/3290605.3300789.

    Book  Google Scholar 

  78. C. L. Corritore, B. Kracher, S. Wiedenbeck. On-line trust: Concepts, evolving themes, a model. International Journal of Human-computer Studies, vol. 58, no. 6, pp. 737–758, 2003. DOI: https://doi.org/10.1016/S1071-5819(03)00041-7.

    Article  Google Scholar 

  79. B. F. Malle, D. Ullman. Chapter 1 — A multidimensional conception and measure of human-robot trust. Trust in Human-robot Interaction, C. S. Nam, J. B. Lyons, Eds., Academic Press (Elsevier), pp. 3–25, 2021. DOI: https://doi.org/10.1016/B978-0-12-819472-0.00001-0.

  80. J. Lee, N. Moray. Trust, control strategies and allocation of function in Homno-mchgiee sytee-ms. Ergonomics, vol. 35, no. 10, pp. 1243–1270, 1992. DOI: https://doi.org/10.1080/00140139208967392.

    Article  Google Scholar 

  81. A. Bussone, S. Stumpf, D. O’Sullivan. The role of explanations on trust and reliance in clinical decision support systems. In Proceedings of International Conference on Healthcare Informatics, IEEE, Dallsr, USA, pp. 160–169, 2015. DOI: https://doi.org/10.1109/ICHI.2015.26.

    Google Scholar 

  82. D. H. Kim, E. Hoque, M. Agrawala. Answering questions about charts and generating visual explanations. In Proceedings of CHI Conference on Human Factors in Computing Systems, ACM, Honolulu, USA, Article number 340, 2020. DOI: https://doi.org/10.1145/3313831.3376467.

    Book  Google Scholar 

  83. A. Balayn, N. Rikalo, C. Lofi, J. Yang, A. Bozzon. How can explainability methods be used to support bug identification in computer vislon model?? In Proceedings of CHI Conference on Human Factors in Computing Systems, ACM, New Oikann, USA, Attide number 184, 2022. DOI: https://doi.org/10.1145/3491102.3517474.

    Google Scholar 

  84. J. Dodge, Q. V. Liao, Y. F. Zhang, R. K. E. Bellamy, C. Dugan. Explaining models: An empirical study of how explanations Impact fairness judgment. In Proceedings of the 24th International Conference on Intelligent User Interfaces, ACM, Marina del Ray, USA, pp. 275–285, 2019. DOI: https://doi.org/10.1145/3301275.3302310.

    Google Scholar 

  85. B. P. Knijnenburg, M. C. Willemsen, Z. Gantner, H. Soncu, C. Newell. Explaining the user experience of recommender systems. User Modeling and User-adapted Interaction, vol. 22, no. 4, pp. 441–504, 2012. DOI: https://doi.org/10.1007/s11257-011-9118-4.

    Article  Google Scholar 

  86. G. Harrison, J. Hanson, C. Jacinto, J. Ramirez, B. Ur. An empirical study on the perceived fairness of realistic, imperfect machine learning models. In Proceedings of Conference on Fairness, Accountability, and Transparency, ACM, Barcelona, Spain, pp. 392–402, 2020. DOI: https://doi.org/10.1145/3351095.3372831.

    Google Scholar 

  87. N. Cggcc-maca, E. M. Redmües, K. P. Gummadi, A. Weller. Human perceptions of fairness in algorithmic decision making: A case study of criminal risk prediction. In Proceedings of World Wide Web Conference, international World Wide Web Conferences Steering Committee, Lyon, France, pp. 903–912, 2018. DOI: https://doi.org/10.1145/3178876.3186138.

  88. N. Tintarev, J. Masthoff. Explaining recommendations: Design and evaluation. Recommender Systems Handbook, F. Ricci, L. Rokach, B. Shapira, Eds., Boston, USA: Springer, Article number 497, 2015. DOI: https://doi.org/10.1007/978-1-4899-7637-6_10.

    Google Scholar 

  89. A. Smith-Renner, R. Fan, M. Birchfield, T. S. Wu, J. Boyd-Graber, D. S. Weld, L. Findlater. No explainability without accountability: An empirical study of explanations and feedback in interactive ml. In Proceedings of CHI Conference on Human Factors in Computing Systems, ACM, Honolulu, USA, Article number 497, 2020. DOI: https://doi.org/10.1145/3313831.3376624.

    Book  Google Scholar 

  90. A. Smith-Renner, V. Kumar, J. Boyd-Graber, K. Seppi, L. Findlater. Digging into user control: Perceptions of adherence and instability in transparent models. In Proceedings of the 25th International Conference on Intelligent User Interfaces, ACM, Cagliari, Italy, pp. 519–530, 2020. DOI: https://doi.org/10.1145/3377325.3377491.

    Google Scholar 

  91. C. Panigutti, A. Beretta, F. Giannotti, D. Pedreschi. Understanding the impact of explanations on advice-taking: A user study for AI-based climcal decision support systems. in Proceedings of CHI Conference on Human Factors in Computing Systems, ACM, New Orleans, USA, Article number 568, 2022. DOI: https://doi.org/10.1145/3491102.3502104

    Google Scholar 

  92. M. M. Fan, X. Y. Yang, T. Yu, Q. V. Liao, J. Zhao Human-AI collaboration for UX evatoation: Effects of explanation and synchronization Proceedings of the ACM on Human-computer Interaction, vol 6, no CSCW1, Article number 96, 2022 DOI: 10 1145/3512943

  93. R. R. Paleja, M. Ghuy, N. R. Arachchige, R. Jensen, M. C. Gombolay The utility of explainable AI in ad hoc human-machine teaming In Proceedings of the 35th Advances in Neural Information Processing Systems, NeurIPS, New Orleans, USA, pp. 610–623, 2022.

    Google Scholar 

  94. A. Kohli, S. Jha. Why cad failed in mammography. Journal of the American College of Radiology, vol. 15, no. 3, pp. 535–537, 2018. DOI: https://doi.org/10.1016/j.jacr.2017.12.029.

    Article  Google Scholar 

  95. V. Lai, H. Liu, C. H. Tan. “Why is ‘Chicago’ deceptive?” towards building model-driven tutorials for humans. In Proceedings of CHI Conference on Human Factors in Computing Systems, ACM, Honolulu, USA, Article number 744, 2020. DOI: https://doi.org/10.1145/3313831.3376873.

    Book  Google Scholar 

  96. J. T. Springenberg, A. Dosovitskiy, T. Brox, M. A. Riedmiller. Striving for simplicity: The all convolutional net. In Proceedings of the 3rd International Conference on Learning Representations, San Diego, USA, 2015. DOI: arxiv.org/abs/1412.6806.

  97. S. Srinivas, F. Fleuret. Full-gradient representation for neural network visualization. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, Article number 371, 2019.

  98. M. Sundararajan, A. Taly, Q. Q. Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, pp. 3319–3328, 2017.

  99. L. Richter, A. Boustati, N. Nüsken, F. J. R. Ruiz, Ö. D. Akyildiz. VarGrad: A low-variance gradient estimator for variational inference. In Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, pp. 13481–13492, 2020.

  100. G. Montavon, S. Lapuschkin, A. Binder, W. Samek, K. R. Müller. Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognition, vol. 65, pp. 211–222, 2017. DOI: https://doi.org/10.1016/j.patcog.2016.11.008.

    Article  Google Scholar 

  101. S. Bach, A. Binder, G. Montavon, F. Klauschen, K. R. Müller, W. Samek. On pixel-wise explanations for nonlinear classifier decisions by layer-wise relevance propagation. PLoS One, vol. 10, no. 7, Article number e0130140, 2015. DOI: https://doi.org/10.1371/journal.pone.0130140.

  102. A. Shrikumar, P. Greenside, A. Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, pp. 3145–3153, 2017.

  103. Y. LeCun, C. Cortes, C. J. C. Burges. The MNIST database of handwritten digits, 1998. [Online], Available: http://yann.lecun.com/exdb/mnist/, Dec. 13, 2022.

  104. A. Krichvvsky. Learning Multiple Layerss of Features from Tiny Images, Technical Report TR-009V, University of Toronto, Canada, 2009.

    Google Scholar 

  105. J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami, USA, pp. 248–255, 2009. DOI: https://doi.org/10.1109/CVPR.2009.5206848.

  106. C. Wah, S. Branson, P. Welinder, P. Perona, S. Belongie. The Caltech-UCSD Birds-200–2011 Dataset. California Institute of Technology, Califoenia, USA, 2011.

    Google Scholar 

  107. M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, A Zisserman The PASCAL visual object classes challenge: A retrospective. International Journal of Computer Vision, vol. 111, no. 1, pp. 98–136, 2015. DOI: https://doi.org/10.1007/s11263-014-0733-5.

    Article  Google Scholar 

  108. A. Chattopadhay, A. Sarkar, P. Howlader, V. N. Balasubramanian. Grad-CAM++: Generaiized gradeent-based visual explanations for deep convolutional networks. In Proceedings of IEEE Winter Conference on Applications of Computer Vision, Lake Tahoe, USA, pp. 839–847, 2018. DOI: https://doi.org/10.1109/WACV.2018.00097.

  109. H. F. Wang, Z. F. Wang, M. N. Du, F. Yang, Z. J. Zhang, S. R. Ding, P. Mardziel, X. Hu. Score-CAM: Score-weighted visual explanations for convolutional neural networks. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, IEEE, Seattle, USA, pp. 111–119, 2020. DOI: https://doi.org/10.1109/CVPRW50498.2020.00020.

    Google Scholar 

  110. M. N. Du, N. H. Liu, Q. Q. Song, X. Hu. Towards explanation of DNN-based prediction with guided feature inversion. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, London, UK, pp. 1358–1367, 2018. DOI: https://doi.org/10.1145/3219819.3220099.

    Google Scholar 

  111. B. L. Zhou, A. Lapedriza, A. Khosla, A. Oliva, A. Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 6, pp. 1452–1464, 2018. DOI: https://doi.org/10.1109/TPAMI.2017.2723009.

    Article  Google Scholar 

  112. B. W. Pan, R. Panda, Y. F. Jiang, Z. Y. Wang, R. Feris, A. Oliva. IA-RED.2: Interpret ability-aware redundancy reduction for vision transformers. In Proceedings of the 35th Annual Conference on Neural Information Processing Systems, pp. 24898–24911, 2021.

  113. H. Chefer, S. Gur, L. Wolf. Transformer interpret ability beyond attention visualization. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Nashville, USA, pp. 782–791, 2021. DOI: https://doi.org/10.1109/CVPR46437.2021.00084.

    Google Scholar 

  114. D. H. Park, L. A. Hendricks, Z. Akata, A. Rohrbach, B. Schiele, T. Darreil, M. Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp.8779–8788, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00915.

    Google Scholar 

  115. H. Chefer, S. Gur, L. Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Montreal, Canada, pp. 387–396, 2021. DOI: https://doi.org/10.1109/ICCV48922.2021.00045.

    Google Scholar 

  116. M. Andriluka, L. Pishchulin, P. Gehler, B. Schiele. 2D human pose estimation: New benchmark and state of the art analysis. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Columbus, USA, pp. 3686–3693, 2014. DOI: https://doi.org/10.1109/CVPR.2014.471.

  117. S. Antol, A. Agrawal, J. S. Lu, M. Mitchell, D. Batra, C. L. Zitnick, D. Parikh. VQA: Visual question answering. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Santiago, Chile, pp. 2425–2433, 2015. DOI: https://doi.org/10.1109/ICCV.2015.279.

    Google Scholar 

  118. J. Carreira, A. Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, pp.4724-733, 2017. DOI: https://doi.org/10.1109/CVPR.2017.502.

  119. S. Carter, Z. Armstrong, L. Schubert, I. Johnson, C. Olah. Activation atlas. Distill, vol.4, no. 3, Article number e15, 2019.

  120. D. Erhan, Y. Bengio, A. Courville, P. Vincent. Visualizing Higher-Layer Features of a Deep Network, Technical Report PDP, Department of Informatics and Operational Research, University of Montreal, Montreal, Canada, 2009.

    Google Scholar 

  121. A. Mahendran, A. Vedaldi. Understanding deep image representations by inverting them. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, pp. 5188–5196, 2015. DOI: https://doi.org/10.1109/CVPR.2015.7299155.

  122. F. Wang, H. J. Liu, J. Cheng. Visualizing deep neural network by alternately image blurring and deblurring. Neural Networks, vol. 97, pp. 162–172, 2018. DOI: https://doi.org/10.1016/j.neunet.2017.09.007.

    Article  Google Scholar 

  123. K. S. Gurumoorthy, A. Dhurandhar, G. Cecchi, C. Aggarwal. Efficient data representation by selecting prototypes with importance weights. In Proceedings of IEEE International Conference on Data Mining, Beijing, China, pp. 260–269, 2019. DOI: https://doi.org/10.1109/ICDM.2019.00036.

  124. C. F. Chen, O. Li, C. F. Tao, A. J. Barnett, J. Su, C. Rudin. This looks like that: Deep learning for interprétable image recognition. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, pp. 8930–8941, 2019.

  125. M. Nauta, R. Van Bree, C. Seifert. Neural prototype trees for interprétable fine-grained image recognition. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, NashviUe, USA, pp. 14928–14938, 2021. DOI: https://doi.org/10.1109/CVPR46437.2021.01469.

    Google Scholar 

  126. J. Krause, M. Stark, J. Deng, F. Li. 3D object representations for fine-grained categorization. In Proceedings of IEEE International Conference on Computer Vision Workshops, Sydney, Australia, pp. 554–561, 2013. DOI: https://doi.org/10.1109/ICCVW.2013.77.

  127. P. W. Koh, P. Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, vol. 70, pp. 1885–1894, 2017.

  128. C. K. Yeh, J. S. Kim, I. E. H. Yen, P. Ravikumar. Representer point selection for explaining deep neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montreal, Canada, pp. 9311–9321, 2018.

  129. J. Crabbé, Z. Z. Qian, F. Imrie, M. Van Der Schaar. Explaining latent representations with a corpus of examples. In Proceedings of the 35th Advances in Neural Information Processing Systems, pp. 12154–12166, 2021.

  130. G. Cohen, S. Afshar, J. Tapson, A. Van Schaik. EMNIST: Extending MNIST to handwritten letters. In Proceedings of International Joint Conference on Neural Networks, IEEE, Anchorage, USA, pp. 2921–2926, 2017. DOI: https://doi.org/10.1109/IJCNN.2017.7966217.

    Google Scholar 

  131. Y. Q. Xian, C. H. Lampert, B. Schiele, Z. Akata. Zero-shot learning- A comprehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 9, pp. 2251–2265, 2019. DOI: https://doi.org/10.1109/TPAMI.2018.2857768.

    Article  Google Scholar 

  132. N. Frosst, G. E. Hinton. Distilling a neural network into a soft decision tree. In Proceedings of the 1st International Workshop on Comprehensibility and Explanation in AI and ML Co-located with 16th International Conference of the Italian Association for Artificial Intelligence, Bari, Italy, 2017. DOI: https://doi.org/10.48550/arXiv.1711.09784.

  133. X. Liu, X. G. Wang, S. Matwin. Improving the interpretability of deep neural networks with knowledge distillation. In Proceedings of IEEE International Conference on Data Mining Workshops, Singapore, pp.905-912, 2018. DOI https://doi.org/10.1109/ICDMW.2018.00132.

  134. Q. S. Zhang, Y. Yang, H. T. Ma, Y. N. Wu. Interpreting CNNs via decision trees. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Long Beach, USA, pp. 6254–6263, 2019. DOI: https://doi.org/10.1109/CVPR.2019.00642.

    Google Scholar 

  135. B. L. Zhou, Y. Y. Sun, D. Bau, A. Torralba. Interprétable basis decomposition for visual explanation. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 122–138, 2018. DOI: https://doi.org/10.1007/978-3-030-01237-3_8.

    Google Scholar 

  136. D. Bau, B. L. Zhou, A. Khosla, A. Oliva, A. Torralba. Network dissection: Quantifying interpret ability of deep visual representations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, pp. 3319–3327, 2017. DOI: https://doi.org/10.1109/CVPR.2017.354.

  137. R. Fong, A. Vedaldi. Net2Vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 8730–8738, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00910.

    Google Scholar 

  138. A. Wan, L. Dunlap, D. Ho, J. Yin, S. Lee, H. Jin, S. Petryk, S. A. Bargal, J. E. Gonzalez. NBDT: Neural-backed decision trees. In Proceedings of the 9th International Conference on Learning Representations, Austria, 2021.

  139. C. Z. Mao, A. Cha, A. Gupta, H. Wang, J. F. Yang, C. Vondrick. Generative interventions for causal learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Nashville, USA, pp. 3946–3955, 2021. DOL 10.1109/CVPR46437.2021.00394.

    Google Scholar 

  140. W. Zhang, X. Zhang, H. W. Deng, M. L. Zhang. Multi-instance causal representation learning for instance label prediction and out-of-distribution generalization. In Proceedings of 36th Conference on Neural Information Processing Systems, New Orleans, USA, 2022.

  141. J. Brehmer, P. de Haan, P. Lippe, T. Cohen. Weakly supervised causal representation learning. In Proceedings of Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems, New Orleans, USA, 2022.

  142. M. Y. Yang, F. R. Liu, Z. T. Chen, X. W. Shen, J. Y. Hao, J. Wang. CausalVAE: Disentangled representation learning via neural structural causal models. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, NashviUe, USA, pp. 9588–9597, 2021. DOI: https://doi.org/10.1109/CVPR46437.2021.00947.

    Google Scholar 

  143. J. Mitrovic, B. McWilliams, J. C. Walker, L. H. Buesing, C. Blundell. Representation learning via invariant causal mechanisms. In Proceedings of the 9th International Conference on Learning Representations, Austria, 2021.

  144. P. Schwab, W. Karlen. CXPlain: Causal explanations for model interpretation under uncertainty. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, pp.10220-10230, 2019.

  145. A. Kori, B. Glocker, F. Toni. GLANCE: Global to local architect ure-neutral concept-based explanations. [Online], Available: https://arxiv.org/abs/2207.01917,2022.

  146. Z. W. Liu, P. Luo, X. G. Wang, X. O. Tang. Deep learning face attributes in the wild. In Proceedings of IEEE International Conference on Computer Vision, Santiago, Chile, pp. 3730–3738, 2015. DOI: https://doi.org/10.1109/ICCV.2015.425.

  147. B. Caputo, H. Müller, J. Martinez-Gomez, M. Villegas, B. Acar, N. Patricia, N. Marvasti, S. Üsküdarli, R. Paredes, M. Cazorla, I. Garcia-Varea, V. Morell. ImageCLEF: Overview and analysis of the results. In Proceedings of the 5th International Conference of the Cross-Language Evaluation Forum for European Languages, Springer, Sheffield, UK, pp. 192–211, 2014. DOI: https://doi.org/10.1007/978-3-319-11382-118.

    Google Scholar 

  148. D. Li, Y. X. Yang, Y. Z. Song, T. M. Hospedales. Deeper, broader and artier domain generalization. In Proceedings of IEEE International Conference on Computer Vision, Venice, Italy, pp. 5543–5551, 2017. DOI: https://doi.org/10.1109/ICCV.2017.591.

  149. X. Yang, H. W. Zhang, G. J. Qi, J. F. Cai. Causal attention for vision-language tasks. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Nashville, USA, pp. 9842–9852, 2021. DOI: https://doi.org/10.1109/CVPR46437.2021.00972.

    Google Scholar 

  150. G. S. Nan, R. Qiao, Y. Xiao, J. Liu, S. C. Leng, H. Zhang, W. Lu. Interventional video grounding with dual contrastive learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Nashville, USA, pp. 2764–2774, 2021. DOI: https://doi.org/10.1109/CVPR46437.2021.00279.

    Google Scholar 

  151. D. Zhang, H. W. Zhang, J. H. Tang, X. S. Hua, Q. R. Sun. Causal intervention for weakly-sup er vised semantic segmentation. In Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, Article number 56, 2020.

  152. T. Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, C. L. Zitnick. Microsoft COCO: Common objects in context. In Proceedings of the 13th European Conference on Computer Vision, Springer, Zurich, Switzerland, pp. 740–755, 2014. DOI: https://doi.org/10.1007/978-3-319-10602-148.

    Google Scholar 

  153. Y. He, Z. Y. Shen, P. Cui. Towards non-I.I.D. Image classification: A dataset and baselines. Pattern Recognition, vol.110, Article number 107383, 2021. DOI: https://doi.org/10.1016/j.patcog.2020.107383.

  154. M. Rohrbach, M. Regneri, M. Andriluka, S. Amin, M. Pinkal, B. Schiele. Script data for attribute-based recognition of composite activities. In Proceedings of the 12th European Conference on Computer Vision, Springer, Florence, Italy, pp. 144–157, 2012. DOI: https://doi.org/10.1007/978-3-642-33718-511.

    Google Scholar 

  155. J. Y. Gao, C. Sun, Z. H. Yang, R. Nevatia. TALL: Temporal activity localization via language query. In Proceedings of IEEE International Conference on Computer Vision, Venice, Italy, pp. 5277–5285, 2017. DOI: https://doi.org/10.1109/IC-CV.2017.563.

  156. F. C. Heilbron, V. Escorcia, B. Ghanem, J. C. Niebles. ActivityNet: A large-scale video benchmark for human activity understanding. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, pp. 961–970, 2015. DOI: https://doi.org/10.1109/CVPR.2015.7298698.

  157. A. Dhurandhar, P. Y. Chen, R. Luss, C. C. Tu, P. Ting, K. Shanmugam, P. Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montreal, Canada, pp. 590–601, 2018.

  158. R. Luss, P. Y. Chen, A. Dhurandhar, P. Sattigeri, Y. F. Zhang, K. Shanmugam, C. C. Tu. Leveraging latent features for local explanations. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Singapore, pp. 1139–1149, 2021. DOI: https://doi.org/10.1145/3447548.3467265.

  159. Y. Goyal, Z. Y. Wu, J. Ernst, D. Batra, D. Parikh, S. Lee. Count er factual visual explanations. In Proceedings of the 36th International Conference on Machine Learning, Long Beach, USA, pp. 2376–2384, 2019.

  160. C. H. Chang, E. Creager, A. Goldenberg, D. Duvenaud. Explaining image classifiers by count er factual generation. In Proceedings of the 7th International Conference on Learning Representations, New Orleans, USA, 2019.

  161. T. Vermeire, D. Brughmans, S. Goethals, R. M. B. De Oliveira, D. Martens. Explainable image classification with evidence counterfactual. Pattern Analysis and Applications, vol. 25, no. 2, pp. 315–335, 2022. DOI: https://doi.org/10.1007/s10044-021-01055-y.

    Article  Google Scholar 

  162. S. Khorram, F. X. Li. Cycle-consistent counterfactuals by latent transformations. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, New Orleans, USA, pp. 10193–10202, 2022. DOI: https://doi.org/10.1109/CVPR52688.2022.00996.

    Google Scholar 

  163. S. Janny, F. Baradel, N. Neverova, M. Nadri, G. Mori, C. Wolf. Filtered-CoPhy: Unsupervised learning of counter-factual physics in pixel space. In Proceedings of the 10th International Conference on Learning Representations, 2022.

  164. P. Jacob, É. Zablocki, H. Ben-Younes, M. Chen, P. Perez, M. Cord. STEEX: Steering counterfactual explanations with semantics. In Proceedings of the 17th European Conference on Computer Vision, Springer, Tel Aviv, Israel, pp. 387–403, 2022. DOI: https://doi.org/10.1007/978-3-031-19775-823.

    Google Scholar 

  165. A. Abid, M. Yüksekgönül, J. Zou. Meaningfully debugging model mistakes using conceptual counterfactual explanations. In Proceedings of the 39th International Conference on Machine Learning, Baltimore, USA, pp.66-88, 2022.

  166. N. C. F. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. KaUoo, K. Liopyris, N. Mishra, H. Kittler, A. Halpern. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC). In Proceedings of the 15th IEEE International Symposium on Biomedical Imaging, Washington DC, USA, pp. 168–172, 2018. DOI: https://doi.org/10.1109/ISBI.2018.8363547.

  167. A. Di Martino, C. G. Yan, Q. Li, E. Denio, F. X. Castellanos, K. Alaerts, J. S. Anderson, M. Assaf, S. Y. Bookheimer, M. Dapretto, B. Deen, S. Delmonte, I. Dinstein, B. Ertl-Wagner, D. A. Fair, L. Gallagher, D. P. Kennedy, C. L. Keown, C. Keysers, J. E. Lainhart, C. Lord, B. Luna, V. Menon, N. J. Minshew, C. S. Monk, S. MueUer, R. A. Millier, M. B. Nebel, J. T. Nigg, K. O’hearn, K. A. Pelphrey, S. J. Peltier, J. D. Rudie, S. Sunaert, M. Thioux, J. M. Tyszka, L. Q. Uddin, J. S. Verhoeven, N. Wenderoth, J. L. Wiggins, S. H. Mostofsky, M. P. Milham. The autism brain imaging data exchange: Towards a large-scale evaluation of the intrinsic brain architecture in autism. Molecular Psychiatry, vol. 19, no. 6, pp. 659–667, 2014. DOI: https://doi.org/10.1038/mp.2013.78.

    Article  Google Scholar 

  168. J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, B. Kim. Sanity checks for saliency maps. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montreal, Canada, pp.9525-9536, 2018.

  169. M. Ghassemi, L. Oakden-Rayner, A. L. Beam. The false hope of current approaches to explainable artificial intelligence in health care. The Lancet Digital Health, vol. 3, no. 11, pp. E745–E750, 2021. DOI: https://doi.org/10.1016/S2589-7500(21)00208-9.

    Article  Google Scholar 

  170. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L Kaiser, I. Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, pp. 6000–6010, 2017.

  171. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. H. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the 9th International Conference on Learning Representations, Austria, 2021.

  172. Z. Liu, Y. T. Lin, Y. Cao, H. Hu, Y. X. Wei, Z. Zhang, S. Lin, B. N. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Montreal, Canada, pp. 9992–10002, 2021. DOI: https://doi.org/10.1109/ICCV48922.2021.00986.

    Google Scholar 

  173. B. H. M. Van Der Velden, H. J. Kuijf, K. G. A. Gilhuijs, M. A. Viergever. Explainable artificial intelligence (XAI) in deep learning-based medical image analysis. Medical Image Analysis, vol.79, Article number 102470, 2022. DOI: https://doi.org/10.1016/j.media.2022.102470.

  174. C. J. Cai, J. Jongejan, J. Holbrook. The effects of example-based explanations in a machine learning interface. In Proceedings of the 24th International Conference on Intelligent User Interfaces, ACM, Marina del Ray, USA, pp. 258–262, 2019. DOI: https://doi.org/10.1145/3301275.3302289.

    Google Scholar 

  175. G. Futia, A. Vetrò. On the integration of knowledge graphs into deep learning models for a more comprehensible Al-three challenges for future research. Information, vol.11, no. 2, Article number 122, 2020. DOI: https://doi.org/10.3390/info11020122.

  176. Y. Rui, V. I. S. Carmona, M. Pourvali, Y. Xing, W. W. Yi, H. B. Ruan, Y. Zhang. Knowledge mining: A cross-disciplinary survey. Machine Intelligence Research, vol. 19, no. 2, pp. 89–114, 2022. DOI: https://doi.org/10.1007/s11633-022-1323-6.

    Article  Google Scholar 

  177. T. Wang, C. Zhou, Q. R. Sun, H. W. Zhang. Causal attention for unbiased visual recognition. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Montreal, Canada, pp. 3071–3080, 2021. DOI: https://doi.org/10.1109/ICCV48922.2021.00308.

    Google Scholar 

  178. P. Spirtes, C. Glymour, R. Schemes. Causation, Prediction, and Search, New York, USA: Springer, 1993. DOI: https://doi.org/10.1007/978-1-4612-2748-9.

    Book  Google Scholar 

  179. W. Burns. Spurious correlations. Retrieved February, vol. 1, Article number 2005, 1997.

  180. E. Bareinboim, J. D. Correa, D. Ibeling, T. Icard. On pearl’s hierarchy and the foundations of causal inference. Probabilistic and Causal Inference: The Works of Judea Pearl. New York, USA: Association for Computing Machinery, pp.507–556, 2022.

    Google Scholar 

  181. J. Pearl. Causal diagrams for empirical research. Biometrika, vol. 82, no. 4, pp. 669–688, 1995. DOI: https://doi.org/10.1093/biomet/82.4.669.

    Article  MathSciNet  Google Scholar 

  182. J. Pearl. Causality: Models, Reasoning, and Inference, 2nd ed., New York, USA: Cambridge University Press, 2009.

    Book  Google Scholar 

  183. J. Pearl, D. Mackenzie. The Book of Why: The New Science of Cause and Effect, New York, USA: Basic Books, Inc., 2018.

    Google Scholar 

  184. A. R. Nogueira, J. Gama, C. A. Ferreira. Causal discovery in machine learning: Theories and applications. Journal of Dynamics and Games, vol. 8, no. 3, pp. 203–231, 2021. DOI: https://doi.org/10.3934/jdg.2021008.

    Article  MathSciNet  Google Scholar 

  185. B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, Y. Bengio. Toward causal representation learning. Proceedings of the IEEE, vol. 109, no. 5, pp. 612–634, 2021. DOI: https://doi.org/10.1109/JPROC.2021.3058954.

    Article  Google Scholar 

  186. Y. Liu, Y. S. Wei, H. Yan, G. B. Li, L. Lin. Causal reasoning meets visual representation learning: A prospective study. Machine Intelligence Research, vol. 19, no. 6, pp. 485–511, 2022. DOI: https://doi.org/10.1007/s11633-022-1362-z.

    Article  Google Scholar 

  187. S. Beckers. Causal explanations and XAI. In Proceedings of the 1st Conference on Causal Learning and Reasoning, Eureka, USA, pp. 90–109, 2022.

  188. C. Molnar, G. Casalicchio, B. Bischl. Interprétable machine learning — A brief history, state-of-the-art and challenges. In Proceedings of EC ML PKDD 2020 Workshops, Ghent, Belgium, pp.417-431, 2020. DOI: https://doi.org/10.1007/978-3-030-65965-328.

  189. Z. Papanastasopoulos, R. K. Samala, H. P. Chan, L. Hadjiiski, C. Paramagul, M. A. Helvie, C. H. Neal. Explainable AI for medical imaging: Deep-learning CNN ensemble for classification of estrogen receptor status from breast MRI. In Proceedings of SPIE 11314, Medical Imaging Computer-aided Diagnosis, Houston, USA, Artide number 113140Z, 2020. DOI: https://doi.org/10.1117/12.2549298.

  190. N. Alwarasneh, Y. S. S. Chow, S. T. M. Yan, C. H. Lim. Bridging explainable machine vision in CAD systems for lung cancer detection. In Proceedings of the 13th International Conference on Intelligent Robotics and Applications, Springer, Kuala Lumpur, Malaysia, pp. 254–269, 2020. DOI: https://doi.org/10.1007/978-3-030-66645-3_22.

    Google Scholar 

  191. Y. Yamamoto, T. Tsuzuki, J. Akatsuka, M. Ueki, H. Morikawa, Y. Numata, T. Takahara, T. Tsuyuki, K. Tsutsumi, R. Nakazawa, A. Shimizu, I. Maeda, S. Tsuchiya, H. Kanno, Y. Kondo, M. Fukumoto, G. Tamiya, N. Ueda, G. Kimura. Automated acquisition of explainable knowledge from unannotated hist opat hology images. Nature Communications, vol.10, no. 1, Article number 5642, 2019. DOI: https://doi.org/10.1038/s41467-019-13647-8.

  192. J. R. Clough, I. Oksuz, E. Puyol-Antón, B. Ruijsink, A. P. King, J. A. Schnabel. Global and local interpret ability for cardiac MRI classification. In Proceedings of the 22nd International Conference on Medical Image Computing and Computer-assisted Intervention, Springer, Shenzhen, China, pp. 656–664, 2019. DOI: https://doi.org/10.1007/978-3-030-32251-9_72.

    Google Scholar 

  193. M. F. Goldberg, M. F. Goldberg. Correction to: Response to letter to the editor, “neuroradiologic manifestations of COVID-19: What the emergency radiologist needs to know ”. Emergency Radiology, vol.28, no. 2, Article number 441, 2021. DOI: https://doi.org/10.1007/s10140-021-01901-w.

  194. Z. Q. Tang, K. V. Chuang, C. DeCarli, L. W. Jin, L. Beckett, M. J. Keiser, B. N. Dugger. Interprétable classification of Alzheimer’s disease pathologies with a convolutional neural network pipeline. Nature Communications, vol.10, no.l, Article number 2173, 2019. DOI: https://doi.org/10.1038/s41467-019-10212-l.

  195. S. Lee, J. Lee, J. Lee, C. K. Park, S. Yoon. Robust tumor localization with pyramid Grad-CAM. [Online], Available: https://arxiv.org/abs/1805.11393, 2018.

  196. N. Arun, N. Gaw, P. Singh, K. Chang, M. Aggarwal, B. Chen, K. Hoebei, S. Gupta, J. Patel, M. Gidwani, J. Adebayo, M. D. Li, J. Kalpathy-Cramer. Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging. Radiology: Artificial Intelligence, vol.3, no. 6, Article number e200267, 2021. DOI: https://doi.org/10.1148/ryai.2021200267.

  197. Y. Oh, S. Park, J. C. Ye. Deep learning COVID-19 features on CXR using limited training data sets. IEEE Transactions on Medical Imaging, vol. 39, no. 8, pp. 2688–2700, 2020. DOI: https://doi.org/10.1109/TMI.2020.2993291.

    Article  Google Scholar 

  198. J. Wu, B. L. Zhou, D. Peck, S. Hsieh, V. Dialani, L. Mackey, G. Patterson. DeepMiner: Discovering interprétable representations for mammogram classification and explanation. Harvard Data Science Review, vol.3, no.4, 2021. DOI: https://doi.org/10.1162/99608f92.8b81b005.

  199. S. W. Shen, S. X. Han, D. R. Aberle, A. A. Bui, W. Hsu. An interprétable deep hierarchical semantic convolution-al neural network for lung nodule malignancy classification. Expert Systems with Applications, vol.128, pp.84–95, 2019. DOI: https://doi.org/10.1016/j.eswa.2019.01.048.

    Article  Google Scholar 

  200. S. Mertes, T. Huber, K. Weitz, A. Heimerl, E. Andre. G A Nt er factual-count er factual explanations for medical non-experts using generative adversarial learning. Frontiers in Artificial Intelhgence, vol.5, Article number 825565, 2022. DOI: https://doi.org/10.3389/frai.2022.825565.

  201. E. Kim, S. Kim, M. Seo, S. Yoon. XProtoNet: Diagnosis in chest radiography with global and local explanations. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Nashville, USA, pp. 15714–15723, 2021. DOI: https://doi.org/10.1109/CVPR46437.2021.01546.

    Google Scholar 

  202. G. H. Fu, R. Q. Wang, J. Q. Li, M. Vakalopoulou, V. Kalogeiton. Me-NDT: Neural-backed decision tree for visual explainability of deep medical models. In Proceedings of the 4th International MIDL Conference on Medical Imaging with Deep Learning, 2021.

  203. M. A. Gulum, C. M. Trombley, M. Kantardzic. Multiple interpretations improve deep learning transparency for prostate lesion detection. In Proceedings of VLDB Workshop on Data Management and Analytics for Medicine and Healthcare, Springer, pp. 120–137, 2021. DOI: 1007/978-3-030-71055-2_11.

  204. W. Q. Shi, L. Tong, Y. D. Zhu, M. D. Wang. COVID-19 automatic diagnosis with radiographic imaging: Explainable attention transfer deep neural networks. IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 7, pp. 2376–2387, 2021. DOI: https://doi.org/10.1109/JBHI.2021.3074893.

    Article  Google Scholar 

  205. T. Y. Peng, M. Boxberg, W. Weichert, N. Navab, C. Marr. Multi-task learning of a deep k-nearest neighbour network for hist opat hological image classification and retrieval. In Proceedings of the 22nd International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, Shenzhen, China, pp. 676–684, 2019. DOI: https://doi.org/10.1007/978-3-030-32239-7_75.

    Google Scholar 

  206. I. Risso-Gill, H. Legido-Quigley, D. Panteli, M. Mckee. Assessing the role of regulatory bodies in managing health professional issues and errors in Europe. International Journal for Quality in Health Care, vol. 26, no. 4, pp. 348–357, 2014. DOI: https://doi.org/10.1093/intqhc/mzu036.

    Article  Google Scholar 

  207. E. Oikonomou, J. Carthey, C. Macrae, C. Vincent. Patient safety regulation in the NHS: Mapping the regulatory landscape of healthcare. BMJ Open, vol. 9, no. 7, Article number e028663, 2019. DOI: https://doi.org/10.1136/bmjopen-2018-028663.

  208. D. Schneeberger, K. Stöger, A. Holzinger. The European legal framework for medical AI. In Proceedings of the 4th International Cross-domain Conference for Machine Learning and Knowledge Extraction, Springer, Dublin, Ireland, pp. 209–226, 2020. DOI: https://doi.org/10.1007/978-3-030-57321-812.

    Google Scholar 

  209. I. P. De Sousa, M. M. B. R. VeUasco, E. C. Da Silva. Explainable artificial intelligence for bias detection in COVID CT-scan classifiers. Sensors, vol.21, no. 16, Article number 5657, 2021. DOI: https://doi.org/10.3390/s21165657.

  210. V. K. Venugopal, K. Vaidhya, M. Murugavel, A. Chunduru, V. Mahajan, S. Vaidya, D. Mahra, A. Rangasai, H. Mahajan. Unboxing AI — Radiological insights into a deep neural network for lung nodule characterization. Academic Radiology, vol. 27, no. l, pp. 88–95, 2020. DOI: https://doi.org/10.1016/j.acra.2019.09.015.

    Article  Google Scholar 

  211. S. M. Lundberg, S. I. Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Red Hook, USA, pp.4768-4777, 2017.

  212. M. Ancona, E. Ceolini, C. Ö ztireli, M. Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada, 2018.

  213. R. Fong, M. Patrick, A. Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Seoul, Republic of Korea, pp. 2950–2958, 2019. DOI: https://doi.org/10.1109/ICCV.2019.00304.

    Google Scholar 

  214. J. Klaise, A. V. Looveren, G. Vacanti, A. Coca. Alibi explain: algorithms for explaining machine learning models. Journal of Machine Learning Research, vol. 22, pp. 1–7, 2021.

    Google Scholar 

  215. T. Fel, L. Hervier, D. Vigouroux, A. Poche, J. Plakoo, R. Cadene, M. Chalvidal, J. Colin, T. Boissin, L. Bethune, A. Picard, C. Nicodeme, L. Gardes, G. Flandin, T. Serre. Xplique: a deep learning explainability toolbox. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognitio, New Orleans, USA, 2022. DOI: https://doi.org/10.48550/arXiv.2206.04394

  216. A. Abdul, J. Vermeulen, D. D. Wang, B. Y. Lim, M. Kankanhalli. Trends and trajectories for explainable, accountable and intelligible systems: An HCl research agenda. In Proceedings of CHI Conference on Human Factorsin Computing Systems, ACM, Montreal, Canada, Article number 582, 2018. DOI: https://doi.org/10.1145/3173574.3174156.

  217. F. Lécué, B. Abeloos, J. Anetil, M. Bergeron, D. Dalla-Rosa, S. Corbeil-Letourneau, F. Martet, T. Pommellet, L. Salvan, S. Veilleux, M. Ziaeefard. Thaies XAI platform: Adaptable explanation of machine learning systems — A knowledge graphs perspective. In Proceedings of ISWC Satellite Tracks (Posters & Demonstrations, Industry, and Outrageous Ideas) Co-located with 18th International Semantic Web Conference, Auckland, New Zealand, pp.315–316, 2019.

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Nos. 61772111 and 72010107002). The authors would like to thank the anonymous reviewers and editors for their helpful comments on this manuscript.

Author information

Authors and Affiliations

  1. Department of Data Science and Management Engineering, School of Management, Zhejiang University, Hangzhou, 310058, China

    Xiangwei Kong, Shujie Liu & Luhao Zhu

Authors
  1. Xiangwei Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shujie Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luhao Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiangwei Kong.

Ethics declarations

The authors declared that they have no conflicts of interest to this work.

Additional information

Colored figures are available in the online version at https://link.springer.com/journal/11633

Xiangwei Kong received the B.Eng. degree in electrical engineering, the M. Sc. degrees in communication and electronic systems from Harbin Shipbuilding Engineering Institute, China in 1981 and 1988, respectively, and the Ph.D. degree in management science and engineering from Dalian University of Technology, China in 2003. From 2006 to 2007, she was a visiting scholar at Department of Computer Science, Purdue University, USA. From 2014 to 2015, she was a senior research scientist at Department of Computer Science, New York University, USA. She is currently a full professor of Zhejiang University, China, in both Faculty of Data Science and Engineering Management and Faculty of Computer Science and Technology.

Her research interests include trustworthy artificial intelligence, multi-modal computing, big data and business analytics.

Shujie Liu received the B.Eng. degree in management information system from Harbin Institute of Technology, China in 2021. She is currently a Ph.D. degree candidate at School of Management in management science and engineering, Zhejiang University, China.

Her research interests include trustworthy artificial intelligence, unstructured data analysis, big data and business analytics.

Luhao Zhu received the B. Mgt. degree in human resource management from School of Management, Zhejiang University, China in 2021. Currently, he is a master student in management science and engineering at School of Management, Zhejiang University, China.

His research interests include explainable artificial intelligence, adversarial learning, and generative model.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kong, X., Liu, S. & Zhu, L. Toward Human-centered XAI in Practice: A survey. Mach. Intell. Res. (2024). https://doi.org/10.1007/s11633-022-1407-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11633-022-1407-3

Keywords

Search

Navigation