Mahsan Nourani
ABSTRACT
EXplainable Artificial Intelligence (XAI) approaches are used to bring transparency to machine learning and artificial intelligence models, and hence, improve the decision-making process for their end-users. While these methods aim to improve human understanding and their mental models, cognitive biases can still influence a user’s mental model and decision-making in ways that system designers do not anticipate. This paper presents research on cognitive biases due to ordering effects in intelligent systems. We conducted a controlled user study to understand how the order of observing system weaknesses and strengths can affect the user’s mental model, task performance, and reliance on the intelligent system, and we investigate the role of explanations in addressing this bias. Using an explainable video activity recognition tool in the cooking domain, we asked participants to verify whether a set of kitchen policies are being followed, with each policy focusing on a weakness or a strength. We controlled the order of the policies and the presence of explanations to test our hypotheses. Our main finding shows that those who observed system strengths early-on were more prone to automation bias and made significantly more errors due to positive first impressions of the system, while they built a more accurate mental model of the system competencies. On the other hand, those who encountered weaknesses earlier made significantly fewer errors since they tended to rely more on themselves, while they also underestimated model competencies due to having a more negative first impression of the model. Our work presents strong findings that aim to make intelligent system designers aware of such biases when designing such tools.
- Amina Adadi and Mohammed Berrada. 2018. Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI). IEEE Access 6(2018), 52138–52160.Google Scholar
Cross Ref
- Ahmed Alqaraawi, Martin Schuessler, Philipp Weiß, Enrico Costanza, and Nadia Berthouze. 2020. Evaluating saliency map explanations for convolutional neural networks: a user study. In Proceedings of the 25th International Conference on Intelligent User Interfaces. 275–285.Google ScholarDigital Library
- Jonathan Baron. 2000. Thinking and deciding. Cambridge University Press.Google Scholar
- Adrian Bussone, Simone Stumpf, and Dympna O’Sullivan. 2015. The role of explanations on trust and reliance in clinical decision support systems. In 2015 International Conference on Healthcare Informatics. IEEE, 160–169.Google ScholarDigital Library
- Isaac Cho, Ryan Wesslen, Alireza Karduni, Sashank Santhanam, Samira Shaikh, and Wenwen Dou. 2017. The anchoring effect in decision-making with visual analytics. In 2017 IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE, 116–126.Google ScholarCross Ref
- Kennith J. W. Craik. 1943. The Nature of Explanation. Cambridge University Press. Google-Books-ID: EN0TrgEACAAJ.Google Scholar
- Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608(2017).Google Scholar
- David Dunning. 2012. Confidence considered: Assessing the quality of decisions and performance. Social metacognition(2012), 63–80.Google Scholar
- David Gunning. 2017. Explainable artificial intelligence (xai). Defense Advanced Research Projects Agency (DARPA), nd Web 2 (2017), 2.Google Scholar
- Pamela Thibodeau Hardiman, Robert Dufresne, and Jose P. Mestre. 1989. The relation between problem categorization and problem solving among experts and novices. Memory & Cognition 17, 5 (Sep 1989), 627–638. https://doi.org/10.3758/BF03197085Google ScholarCross Ref
- Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608(2018).Google Scholar
- Fred Hohman, Minsuk Kahng, Robert Pienta, and Duen Horng Chau. 2018. Visual analytics in deep learning: An interrogative survey for the next frontiers. IEEE transactions on visualization and computer graphics 25, 8(2018), 2674–2693.Google ScholarDigital Library
- Donald Honeycutt, Mahsan Nourani, and Eric Ragan. 2020. Soliciting human-in-the-loop user feedback for interactive machine learning reduces user trust and impressions of model accuracy. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 8. 63–72.Google ScholarCross Ref
- Philip N. Johnson-Laird. 2010. Mental models and human reasoning. Proceedings of the National Academy of Sciences 107, 43 (Oct 2010), 8. https://doi.org/10.1073/pnas.1012933107Google ScholarCross Ref
- Zafar A. Khan and Won Sohn. 2011. Abnormal human activity recognition system based on R-transform and kernel discriminant technique for elderly home care. IEEE Transactions on Consumer Electronics 57, 4 (Nov 2011), 1843–1850. https://doi.org/10.1109/TCE.2011.6131162Google ScholarCross Ref
- Antino Kim, Mochen Yang, and Jingjng Zhang. 2020. When Algorithms Err: Differential Impact of Early vs. Late Errors on Users’ Reliance on Algorithms. Late Errors on Users’ Reliance on Algorithms (July 2020) (2020).Google Scholar
- Olga Kostopoulou, Miroslav Sirota, Thomas Round, Shyamalee Samaranayaka, and Brendan C Delaney. 2017. The role of physicians’ first impressions in the diagnosis of possible cancers without alarm symptoms. Medical Decision Making 37, 1 (2017), 9–16.Google ScholarCross Ref
- Tai Yu Lai, Jong Yih Kuo, Yong-Yi Fanjiang, Shang-Pin Ma, and Yi Han Liao. 2012. Robust Little Flame Detection on Real-Time Video Surveillance System. In 2012 Third International Conference on Innovations in Bio-Inspired Computing and Applications. 139–143. https://doi.org/10.1109/IBICA.2012.41Google ScholarDigital Library
- Q Vera Liao, Daniel Gruen, and Sarah Miller. 2020. Questioning the AI: Informing Design Practices for Explainable AI User Experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–15.Google ScholarDigital Library
- Geoffrey K Lighthall and Cristina Vazquez-Guillamet. 2015. Understanding decision making in critical care. Clinical medicine & research 13, 3-4 (2015), 156–168.Google Scholar
- Zachary C Lipton. 2018. The mythos of model interpretability. Queue 16, 3 (2018), 31–57.Google ScholarDigital Library
- Stephanie M. Merritt. 2011. Affective Processes in Human–Automation Interactions. Human Factors 53, 4 (Aug 2011), 356–370. https://doi.org/10.1177/0018720811411912Google ScholarCross Ref
- Stephanie M. Merritt and Daniel R. Ilgen. 2008. Not All Trust Is Created Equal: Dispositional and History-Based Trust in Human-Automation Interactions. Human Factors 50, 2 (Apr 2008), 194–210. https://doi.org/10.1518/001872008X288574Google ScholarCross Ref
- Robert K. Merton and Patricia L. Kendall. 1946. The Focused Interview. Amer. J. Sociology 51, 6 (May 1946), 541–557. https://doi.org/10.1086/219886Google ScholarCross Ref
- Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence 267 (2019), 1–38.Google ScholarCross Ref
- Tim Miller, Piers Howe, and Liz Sonenberg. 2017. Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv preprint arXiv:1712.00547(2017).Google Scholar
- Sina Mohseni, Niloofar Zarei, and Eric D Ragan. 2018. A survey of evaluation methods and measures for interpretable machine learning. ACM Transactions on Interactive Intelligent Systems (2018).Google Scholar
- Don A Moore and Paul J Healy. 2008. The trouble with overconfidence.Psychological review 115, 2 (2008), 502.Google Scholar
- Donald A. Norman. 1983. Some Observations on Mental Models(1 ed.). Lawrence Erlbaum Associates Inc. pp7-14, 7–14. https://ar264sweeney.files.wordpress.com/2015/11/norman_mentalmodels.pdfGoogle Scholar
- Mahsan Nourani, Donald R Honeycutt, Jeremy E Block, Chiradeep Roy, Tahrima Rahman, Eric D Ragan, and Vibhav Gogate. 2020. Investigating the importance of first impressions and explainable ai with interactive video analysis. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1–8.Google ScholarDigital Library
- Mahsan Nourani, Joanie King, and Eric Ragan. 2020. The role of domain expertise in user trust and the impact of first impressions with intelligent systems. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 8. 112–121.Google ScholarCross Ref
- Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. 2018. The building blocks of interpretability. Distill 3, 3 (2018), e10.Google ScholarCross Ref
- Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (to appear in the Proceedings of ACM CHI 2021) (2018).Google Scholar
- Tahrima Rahman, Prasanna Kothalkar, and Vibhav Gogate. 2014. Cutset networks: A simple, tractable, and scalable approach for improving the accuracy of Chow-Liu trees. In Joint European conference on machine learning and knowledge discovery in databases. Springer, 630–645.Google ScholarDigital Library
- William E Remus and Jeffrey E Kottemann. 1986. Toward intelligent decision support systems: An artificially intelligent statistician. MIS Quarterly (1986), 403–418.Google Scholar
- Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. ” Why should I trust you?” Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 1135–1144.Google ScholarDigital Library
- Anna Rohrbach, Marcus Rohrbach, Wei Qiu, Annemarie Friedrich, Manfred Pinkal, and Bernt Schiele. 2014. Coherent multi-sentence video description with variable level of detail. In German conference on pattern recognition. Springer, 184–195.Google ScholarCross Ref
- Chiradeep Roy, Mahesh Shanbhag, Mahsan Nourani, Tahrima Rahman, Samia Kabir, Vibhav Gogate, Nicholas Ruozzi, and Eric D Ragan. 2019. Explainable Activity Recognition in Videos.. In IUI Workshops.Google Scholar
- J. Edward Russo, Eric J. Johnson, and Debra L. Stephens. 1989. The validity of verbal protocols. Memory & Cognition 17, 6 (Nov 1989), 759–769. https://doi.org/10.3758/BF03202637Google ScholarCross Ref
- James Schaffer, John O’Donovan, James Michaelis, Adrienne Raglin, and Tobias Höllerer. 2019. I can do better than your AI: Expertise and explanations. In Proceedings of the 24th International Conference on Intelligent User Interfaces. 240–251.Google ScholarDigital Library
- Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1–9.Google ScholarCross Ref
- Dairazalia Sánchez, Monica Tentori, and Favela Jesús. 2008. Activity Recognition for the Smart Hospital. IEEE Intelligent Systems 23, 02 (Apr 2008), 50–57. https://doi.org/10.1109/MIS.2008.18Google ScholarDigital Library
- Philip E Tetlock. 1983. Accountability and the perseverance of first impressions. Social Psychology Quarterly(1983), 285–292.Google Scholar
- Rajesh Kumar Tripathi, Anand Singh Jalal, and Subhash Chand Agrawal. 2018. Suspicious human activity recognition: a review. Artificial Intelligence Review 50, 2 (Aug 2018), 283–339. https://doi.org/10.1007/s10462-017-9545-7Google ScholarDigital Library
- Emily Wall, Leslie Blaha, Celeste Paul, and Alex Endert. 2019. A formative study of interactive bias metrics in visual analytics using anchoring bias. In IFIP Conference on Human-Computer Interaction. Springer, 555–575.Google ScholarDigital Library
- Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y Lim. 2019. Designing theory-driven user-centric explainable AI. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–15.Google ScholarDigital Library
- David S Watson, Jenny Krutzinna, Ian N Bruce, Christopher EM Griffiths, Iain B McInnes, Michael R Barnes, and Luciano Floridi. 2019. Clinical applications of machine learning algorithms: beyond the black box. Bmj 364(2019).Google Scholar
- Daniel S Weld and Gagan Bansal. 2019. The challenge of crafting intelligible intelligence. Commun. ACM 62, 6 (2019), 70–79.Google ScholarDigital Library
- Serena Yeung, Francesca Rinaldo, Jeffrey Jopling, Bingbin Liu, Rishab Mehra, N. Lance Downing, Michelle Guo, Gabriel M. Bianconi, Alexandre Alahi, Julia Lee, and et al.2019. A computer vision system for deep learning-based detection of patient mobilization activities in the ICU. npj Digital Medicine 2, 11 (Mar 2019), 1–5. https://doi.org/10.1038/s41746-019-0087-zGoogle ScholarCross Ref
Index Terms
- Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems
Index terms have been assigned to the content through auto-classification.
Recommendations
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing SystemsA surge of interest in explainable AI (XAI) has led to a vast collection of algorithmic work on the topic. While many recognize the necessity to incorporate explainability features in AI systems, how to address real-world user needs for understanding AI ...
On the Importance of User Backgrounds and Impressions: Lessons Learned from Interactive AI Applications
While EXplainable Artificial Intelligence (XAI) approaches aim to improve human-AI collaborative decision-making by improving model transparency and mental model formations, experiential factors associated with human users can cause challenges in ways ...
User Trust and Understanding of Explainable AI: Exploring Algorithm Visualisations and User Biases
Human-Computer Interaction. Human Values and Quality of LifeAbstractArtificial intelligence (AI) is increasingly being integrated into different areas of our lives. AI has the potential to increase productivity and relieve workload on staff in high-pressure jobs such as healthcare. However, most AI healthcare ...
Comments