Jennifer Cobbe
ABSTRACT
This paper introduces reviewability as a framework for improving the accountability of automated and algorithmic decisionmaking (ADM) involving machine learning. We draw on an understanding of ADM as a socio-technical process involving both human and technical elements, beginning before a decision is made and extending beyond the decision itself. While explanations and other model-centric mechanisms may assist some accountability concerns, they often provide insufficient information of these broader ADM processes for regulatory oversight and assessments of legal compliance. Reviewability involves breaking down the ADM process into technical and organisational elements to provide a systematic framework for determining the contextually appropriate record-keeping mechanisms to facilitate meaningful review - both of individual decisions and of the process as a whole. We argue that a reviewability framework, drawing on administrative law's approach to reviewing human decision-making, offers a practical way forward towards more a more holistic and legally-relevant form of accountability for ADM.
- Philip Alston. 2019. Report of the Special Rapporteur on extreme poverty and human rights. United Nations General Assembly A/74/493 (2019). https://undocs.org/A/74/493Google Scholar
- Mike Ananny and Kate Crawford. 2016. Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. New Media & Society (2016).Google Scholar
- Matthew Arnold, Rachel KE Bellamy, Michael Hind, Stephanie Houde, Sameep Mehta, A Mojsilović, Ravi Nair, K Natesan Ramamurthy, Alexandra Olteanu, David Piorkowski, et al. 2019. FactSheets: Increasing trust in AI services through supplier's declarations of conformity. IBM Journal of Research and Development 63, 4/5 (2019), 6--1.Google ScholarCross Ref
- Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Richard Benjamins, Raja Chatlia, and Francisco Herrera. 2020. Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI. Information Fusion 58 (2020).Google Scholar
- Adam Bates, Dave (Jing) Tian, Kevin R.B. Butler, and Thomas Moyer. 2015. Trustworthy Whole-System Provenance for the Linux Kernel. In 24th USENIX Security Symposium (USENIX Security 15). USENIX Association, Washington, D.C., 319--334. https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/batesGoogle ScholarDigital Library
- Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics 6 (2018), 587--604.Google ScholarCross Ref
- Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José M. F. Moura, and Peter Eckersley. 2020. Explainable Machine Learning in Deployment. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* '20). Association for Computing Machinery, New York, NY, USA, 648?657. https://doi.org/10.1145/3351095.3375624Google ScholarDigital Library
- Mark Bovens. 2006. Analysing and Assessing Public Accountability. A Conceptual Framework. EUROGOV, European Governance Papers No. C-06-01 (2006).Google Scholar
- Miles Brundage, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian Hadfield, Heidy Khlaaf, Jingying Yang, Helen Toner, Ruth Fong, et al. 2020. Toward trustworthy AI development: mechanisms for supporting verifiable claims. arXiv preprint arXiv:2004.07213 (2020).Google Scholar
- Jenna Burrell. 2016. How the machine 'thinks': Understanding opacity in machine learning algorithms. Big Data & Society 3 (2016). Issue 1.Google Scholar
- Danielle K Citron and Frank Pasquale. 2014. The Scored Society: Due Process for Automated Predictions. University of Maryland Francis King Carey School of Law Legal Studies Research Paper 8 (2014).Google Scholar
- Richard Cloete, Chris Norval, and Jatinder Singh. 2020. A Call for Auditable Virtual, Augmented and Mixed Reality.Google Scholar
- Jennifer Cobbe. 2019. Administrative Law and the Machines of Government: Judicial Review of Automated Public-Sector Decision-Making. Legal Studies (2019).Google Scholar
- Jennifer Cobbe, Michelle Seng Ah Lee, Heleen Janssen, and Jatinder Singh. 2020. Centring the Rule of Law in the Digital State. IEEE Computer (2020).Google Scholar
- Cary Coglianese and Daniel Lehr. 2017. Regulating by Robot: Administrative Decision Making in the Machine-Learning Era. Georgetown Law Journal 105 (2017).Google Scholar
- European Commission. 2020. Denmark AI Strategy Report. https://ec.europa.eu/knowledge4policy/ai-watch/denmark-ai-strategy-report_en#regulationGoogle Scholar
- European Commission. 2020. White Paper On Artificial Intelligence - A European Approach to Excellence and Trust. COM(2020) 65 (2020).Google Scholar
- John Danaher. 2016. Three Types of Algorithmic Opacity. Algocracy and the Transhumanist Project (5 March 2016). https://algocracy.wordpress.com/2016/03/05/three-types-of-algorithmic-opacityGoogle Scholar
- Jeffrey Dastin. 2018. Amazon scraps secret AI recruiting tool that showed bias against women. Reuters (Oct 2018). https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08GGoogle Scholar
- Demos, doteveryone, Global Partners Digital, Institute for Strategic Dialogue. 2020. Algorithm Inspection and Regulatory Access. Joint Paper (2020). https://demos.co.uk/blog/algorithm-inspection-and-regulatory-access/Google Scholar
- Nicholas Diakopolous. [n.d.]. Algorithmic Accountability Reporting: On the Investigation of Black Boxes. Tow Center for Digital Journalism ([n. d.]).Google Scholar
- Nicholas Diakopolous. 2015. Algorithmic Accountability: Journalistic Investigation of Computational Power Structures. Digital Journalism 3 (2015). Issue 3.Google ScholarCross Ref
- Nicholas Diakopolous. 2016. Accountability in Algorithmic Decision Making: A view from computational journalism. Commun. ACM 59 (2016). Issue 2.Google ScholarDigital Library
- Pedro Domingos. 2012. A Few Useful Things to Know about Machine Learning. Commun. ACM 55, 10 (Oct. 2012), 78?87. https://doi.org/10.1145/2347736. 2347755Google ScholarDigital Library
- L. Duboc, S. Betz, B. Penzenstadler, S. Akinli Kocak, R. Chitchyan, O. Leifler, J. Porras, N. Seyff, and C. C. Venters. 2019. Do we Really Know What we are Building? Raising Awareness of Potential Sustainability Effects of Software Systems in Requirements Engineering. In 2019 IEEE 27th International Requirements Engineering Conference (RE). 6--16.Google Scholar
- Lilian Edwards and Michael Veale. 2017. Slave to the Algorithm? Why a 'Right to an Explanation' is Probably Not the Remedy You Are Looking For. Duke Law & Technology Review 17 (2017).Google Scholar
- Avi Feller, Emma Pierson, Sam Corbett-Davies, and Sharad Goel. 2016. A computer program used for bail and sentencing decisions was labeled biased against blacks. It's actually not that clear. The Washington Post (2016).Google Scholar
- Agata Foryciarz, Daniel Leufer, and Katarzyna Szymielewicz. 2020. Black-Boxed Politics: Opacity is a Choice in AI Systems. Panoptykon Foundation (17 January 2020). https://en.panoptykon.org/articles/black-boxed-politics-opacity-choice-ai-systemsGoogle Scholar
- Sorelle A Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2016. On the (im) possibility of fairness. arXiv preprint arXiv:1609.07236 (2016).Google Scholar
- Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010 (2018).Google Scholar
- Bryce Goodman and Seth Flaxman. 2016. European Union regulations on algorithmic decision-making and a 'right to an explanation'. 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016) (2016).Google Scholar
- Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and Fosca Giannotti. 2018. A Survey of methods for Explaining Black Box Models. Comput. Surveys 51 (2018). Issue 5.Google Scholar
- Information and Privacy Commission New South Wales. 2020. Case Summary on Automated decision making and access to information under the GIPA Act. https://www.ipc.nsw.gov.au/case-summary-automated-decision-making-and-access-information-under-gipa-actGoogle Scholar
- Ada Lovelaec Institute. 2020. Examining the Black Box: Tools for Assessing Algorithmic Systems. (2020).Google Scholar
- Heleen L Janssen. 2020. An approach for a fundamental rights impact assessment to automated decision-making. International Data Privacy Law 10, 1 (03 2020), 76--106. https://doi.org/10.1093/idpl/ipz028 arXiv:https://academic.oup.com/idpl/article-pdf/10/1/76/33151837/ipz028.pdfGoogle ScholarCross Ref
- Michael Katell, Meg Young, Dharma Dailey, Bernease Herman, Vivian Guetler, Aaron Tam, Corinne Binz, Daiella Raz, and P M Krafft. 2020. Towards Situated Interventions for Algorithmic Equity: Lessons from the Field. Fat* '20: Proceedings of the 2020 Conference on Fairness, Accountabiltiy and Transparency (2020).Google Scholar
- Jakko Kemper and Daan Kolkman. 2018. Transparent to whom? No algorithmic accountability without a critical audience. Information, Communication & Society 22 (2018). Issue 14.Google Scholar
- Amir E Khandani, Adlar J Kim, and Andrew W Lo. 2010. Consumer credit-risk models via machine-learning algorithms. Journal of Banking & Finance 34, 11 (2010), 2767--2787.Google ScholarCross Ref
- Ansgar Koene, Chris Clifton, Yohko Hatada, Helena Webb, Menisha Patel, Machado Caio, Jack LaViolette, Rashida Richardson, and Dillon Reisman. 2019. A governance framework for algorithmic accountability and transparency. European Parliamentary Research Service, Panel for the Future of Science and Technology PE 624.262 (April 2019).Google Scholar
- SB Kotsiantis, Dimitris Kanellopoulos, and PE Pintelas. 2006. Data preprocessing for supervised leaning. International Journal of Computer Science 1, 2 (2006), 111--117.Google Scholar
- Maciej Kuziemski and Gianluca Misuraca. 2020. AI governance in the public sector: Three tales from the frontiers of automated decision-making in democratic settings. Telecommunications Policy (2020), 101976.Google Scholar
- P.A. Laplante. 2017. Requirements Engineering for Software and Systems, Third Edition. Taylor & Francis. https://books.google.com.au/books?id=XfvnswEACAAJGoogle Scholar
- David Lehr and Paul Ohm. 2017. Playing with the Data: What Legal Scholars Should Learn About Machine Learning. UC Davis Law Review 51 (2017).Google Scholar
- Gianclaudio Malgieri and Giovanni Comandé. 2017. Why a Right to Legibility of Automated Decision-Making Exists in the General Data Protection Regulation. International Data Privacy Law 7 (2017). Issue 4.Google ScholarCross Ref
- Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency. 220--229.Google ScholarDigital Library
- Dierdre K Mulligan and Kenneth A Bamberger. 2019. Procurement as Policy: Administrative Process for Machine Learning. Berkeley Technology Law Journal 34 (2019).Google Scholar
- Chris Norval, Jennifer Cobbe, and Jatinder Singh. To Appear. Towards an accountable Internet of Things: A call for 'reviewability'. In Privacy by Design for the Internet of Things: Building Accountability and Security. The Institution of Engineering and Technology.Google Scholar
- Government of Canada. 2019. Directive on Automated Decision Making. https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=32592Google Scholar
- Government of Canada. 2020. Algorithmic Impact Assessment. https://open.canada.ca/aia-eia-js/?lang=enGoogle Scholar
- House of Common Science and Technology Committee. 2018. Algorithms in Decision-Making. Fourth Report of Session 2017-19 HC 351 (2018).Google Scholar
- Venice Commission of the Council of Europe. 2016. The Rule of Law Checklist. (2016).Google Scholar
- Information Commissioner's Office. 2020. Guidance on the AI auditing framework. (2020). https://ico.org.uk/for-organisations/guide-to-data-protection/key-data-protection-themes/guidance-on-artificial-intelligence-and-data-protection/Google Scholar
- Information Commissioner's Office and The Alan Turing Institute. 2020. Explaining decisions made with AI. https://ico.org.uk/for-organisations/guide-to-data-protection/key-data-protection-themes/explaining-decisions-made-with-aiGoogle Scholar
- Partnership on AI. 2020. When AI Systems Fail: Introducing the AI Incident Database. https://www.partnershiponai.org/aiincidentdatabase/Google Scholar
- Marion Oswald. 2018. Algorithm-assisted decision-making in the public sector: framing the issues using administrative law rules governing discretionary power. Philosophical Transactions of the Royal Society A 376 (2018).Google Scholar
- Partnership on AI. 2019. Annotation and Benchmarking on Understanding and Transparency of Machine learning Lifecycles (ABOUT ML). https://www.partnershiponai.org/wp-content/uploads/2019/07/ABOUT-ML-v0-Draft-Final.pdfGoogle Scholar
- Frank Pasquale. 2015. The Black Box Society: The Secret Algorithms That Control Money and Information. Harvard University Press.Google Scholar
- Anya ER Prince and Daniel Schwarcz. 2019. Proxy discrimination in the age of artificial intelligence and big data. Iowa L. Rev. 105 (2019), 1257.Google Scholar
- Douglas Pyper. 2020. Research Briefing: The Public Sector Equality Duty and Equality Impact Assessments. House of Commons Library (2020). https://commonslibrary.parliament.uk/research-briefings/sn06591/Google Scholar
- Inioluwa Deborah Radi, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020. Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing. Fat* '20: Proceedings of the 2020 Conference on Fairness, Accountabiltiy and Transparency (2020).Google Scholar
- Dillon Reisman, Jason Schultz, Kate Crawford, and Meredith Whittaker. 2018. Algorithmic Impact Assessments: A practical framework for public agency accountability (AI Now). https://ainowinstitute.org/aiareport2018.pdfGoogle Scholar
- Sebastian Schelter, Joos-Hendrik Böse, Johannes Kirschnick, Thoralf Klein, and Stephan Seufert. 2017. Automatically tracking metadata and provenance of machine learning experiments. In ML Systems Workshop at NIPS.Google Scholar
- Nick Seaver. 2013. Knowing Algorithms. paper presented at Media in Transition 8 (2013).Google Scholar
- Andrew D Selbst and Julia Powles. 2017. Meaningful information and the right to explanation. International Data Privacy Law 7 (2017). Issue 4.Google ScholarCross Ref
- J. Singh and J. Cobbe. 2019. The Security Implications of Data Subject Rights. IEEE Security & Privacy 17, 6 (2019), 21--30.Google ScholarDigital Library
- Jatinder Singh, Jennifer Cobbe, and Chris Norval. 2020. Decision Provenance: Harnessing Data Flow for Accountable Systems. IEEE Access 7 (2020).Google Scholar
- Jatinder Singh, Ian Walden, Jon Crowcroft, and Jean Bacon. 2016. Responsibility & Machine Learning: Part of a Process. Available on SSRN (2016). http://dx.doi.org/10.2139/ssrn.2860048Google Scholar
- Cynthia Stohl, Michael Stohl, and Paul M Leonardi. 2016. Managing Opacity: Information Visibility and the Paradox of Transparency in the Digital Age. International Journal of Communication 10 (2016).Google Scholar
- Harini Suresh and John V Guttag. 2019. A framework for understanding unintended consequences of machine learning. arXiv preprint arXiv:1901.10002 (2019).Google Scholar
- Joe Tomlinson, Katy Sheridan, and Adam Harkens. 2019. Proving Public Law Error in Automated Decision-Making Systems. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3476657Google Scholar
- UK Government. 2020. Guidelines for AI procurement. https://www.gov.uk/government/publications/guidelines-for-ai-procurement/guidelines-for-ai-procurementGoogle Scholar
- European Union. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). Official Journal of the European Union L119 (4 May 2016), 1--88.Google Scholar
- Neil Vigdor. 2019. Apple Card Investigated After Gender Discrimination Complaints. New York Times (Nov 2019). https://www.nytimes.com/2019/11/10/business/Apple-credit-card-investigation.htmlGoogle Scholar
- Sandra Wachter, Brent Mittelstadt, and Luciano Floridi. 2017. Why a Right to Explanation of Automated Decision-Making Does Not Exist in the General Data Protection Regulation. International Data Privacy Law 7 (2017). Issue 2.Google ScholarCross Ref
- Maranke Wieringa. 2020. What to account for when accounting for algorithms: a systematic literature review on algorithmic accountability. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 1--18.Google ScholarDigital Library
- Janis Wong and Tristan Henderson. 2018. How Portable is Portable? Exercising the GDPR's Right to Data Portability. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers (Singapore, Singapore) (UbiComp '18). Association for Computing Machinery, New York, NY, USA, 911--920. https://doi.org/10.1145/3267305.3274152Google ScholarDigital Library
- World Economic Forum. 2020. AI Procurement in a Box: AI Government Procurement Guidelines. http://www3.weforum.org/docs/WEF_AI_Procurement_in_a_Box_AI_Government_Procurement_Guidelines_2020.pdfGoogle Scholar
- Monika Zalnieriute, Lyria Bennett Moses, and George Williams. 2019. The Rule of Law and Automation of Government Decision-Making. Modern Law Review (2019). https://www.modernlawreview.co.uk/may-2019/rule-law-automation-government-decision-makingGoogle Scholar
Index Terms
- Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems
Recommendations
Accountability in Algorithmic Systems: From Principles to Practice
CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing SystemsGrowing concerns over the societal implications of artificial intelligence has motivated an interdisciplinary push towards mechanisms and tools that hold algorithmic systems accountable. Although there have been considerable strides around defining what ...
Just accountability structures – a way to promote the safe use of automated decision-making in the public sector
AbstractThe growing use of automated decision-making (ADM) systems in the public sector and the need to control these has raised many legal questions in academic research and in policymaking. One of the timely means of legal control is accountability, ...
Will Algorithms Blind People? The Effect of Explainable AI and Decision-Makers’ Experience on AI-supported Decision-Making in Government
Computational artificial intelligence (AI) algorithms are increasingly used to support decision making by governments. Yet algorithms often remain opaque to the decision makers and devoid of clear explanations for the decisions made. In this study, we ...
Comments