Skip to main content
SpringerLink
Log in

Managing Bias in Machine Learning Projects

  • Conference paper
  • First Online:
Innovation Through Information Systems (WI 2021)

Part of the book series: Lecture Notes in Information Systems and Organisation ((LNISO,volume 47))

Included in the following conference series:

Abstract

This paper introduces a framework for managing bias in machine learning (ML) projects. When ML-capabilities are used for decision making, they frequently affect the lives of many people. However, bias can lead to low model performance and misguided business decisions, resulting in fatal financial, social, and reputational impacts. This framework provides an overview of potential biases and corresponding mitigation methods for each phase of the well-established process model CRISP-DM. Eight distinct types of biases and 25 mitigation methods were identified through a literature review and allocated to six phases of the reference model in a synthesized way. Furthermore, some biases are mitigated in different phases as they occur. Our framework helps to create clarity in these multiple relationships, thus assisting project managers in avoiding biased ML-outcomes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. von Krogh, G.: Artificial Intelligence in organizations: new opportunities for phenomenon-based theorizing. AMD. 4, 404–409 (2018)

    Article  Google Scholar 

  2. Samuel, A.L.: Some studies in machine learning using the game of checkers. IBM J. Res. Dev. 3, 210–229 (1959)

    Article  Google Scholar 

  3. Sauter, V.L.: Decision Support Systems for Business Intelligence. Wiley, Hoboken (2011)

    Book  Google Scholar 

  4. Berente, N., Gu, B., Santhanam, R., Recker, J.: Call for papers MISQ special issue on managing AI. MISQ (2019)

    Google Scholar 

  5. Mikalef, P., Popovic, A., Eriksson Lundström, J., Conboy, K.: Special issue call for papers: dark side of analytics and AI. Eur. J. Inf. Syst. (2020)

    Google Scholar 

  6. O’Neil, C.: Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Allen Lane, London (2016)

    Google Scholar 

  7. Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51, 1–42 (2018)

    Article  Google Scholar 

  8. Benbya, H., Pachidi, S., Davenport, T., Jarvenpaa, S.: Call for papers: artificial intelligence in organizations: opportunities for management and implications for IS research. J. Assoc. Inf. Syst. (JAIS) MISQ Exec. (MISQE) (2019)

    Google Scholar 

  9. Barocas, S., Selbst, A.D.: Big data’s disparate impact. Calif. Law Rev. 104, 671–732 (2016)

    Google Scholar 

  10. Bailey, D., Faraj, S., Hinds, P., von Krogh, G., Leonardi, P., Hall, P.: Call for papers special issue of organization science: emerging technologies and organizing. Organ. Sci. (2019)

    Google Scholar 

  11. Moreira Nascimento, A., Cortez da Cunha, M.A.V., de Souza Meirelles, F., Scornavacca, E., de Melo, V.V.: A literature analysis of research on artificial intelligence in management information system (MIS). In: AMCIS Proceedings, pp. 1–10 (2018)

    Google Scholar 

  12. Mariscal, G., Marbán, Ó., Fernández, C.: A survey of data mining and knowledge discovery process models and methodologies. Knowl. Eng. Rev. 25, 137–166 (2010)

    Article  Google Scholar 

  13. Martínez-Plumed, F., et al.: CRISP-DM twenty years later: from data mining processes to data science trajectories. IEEE Trans. Knowl. Data Eng., 1–1 (2019)

    Google Scholar 

  14. Chapman, P., et al.: CRISP-DM 1.0: step-by-step data mining guide. SPSS inc. 9, 13 (2000)

    Google Scholar 

  15. Suresh, H., Guttag, J.V.: A framework for understanding unintended consequences of machine learning. http://arxiv.org/abs/1901.10002 (2019)

  16. Baeza-Yates, R.: Bias on the web. Commun. ACM. 61, 54–61 (2018)

    Article  Google Scholar 

  17. Feuerriegel, S., Dolata, M., Schwabe, G.: Fair AI: challenges and opportunities. Bus. Inf. Syst. Eng. 62, 379–384 (2020)

    Article  Google Scholar 

  18. Friedler, S.A., Choudhary, S., Scheidegger, C., Hamilton, E.P., Venkatasubramanian, S., Roth, D.: A comparative study of fairness-enhancing interventions in machine learning. In: Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency, pp. 329–338. ACM, New York (2019)

    Google Scholar 

  19. Friedman, B., Nissenbaum, H.: Bias in computer systems. ACM Trans. Inf. Syst. 14, 330–347 (1996)

    Article  Google Scholar 

  20. Mitchell, S., Potash, E., Barocas, S., D’Amour, A., Lum, K.: Prediction-based decisions and fairness: a catalogue of choices, assumptions, and definitions. http://arxiv.org/abs/1811.07867 (2018)

  21. Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pp. 214–226. Association for Computing Machinery, New York (2011)

    Google Scholar 

  22. Zemel, R., Wu, Y., Swersky, K., Pitassi, T., Dwork, C.: Learning fair representations. In: International Conference on Machine Learning, Atlanta, GA, USA , pp. 325–333. JMLR (2013)

    Google Scholar 

  23. Narayanan, A.: Translation tutorial: 21 fairness definitions and their politics. In: Proceedings of the Conference on Fairness Accountability Transparency, New York, USA, p. 1 (2018)

    Google Scholar 

  24. Green, B., Hu, L.: The myth in the methodology: towards a recontextualization of fairness in machine learning. In: Proceedings of the Machine Learning: the Debates Workshop, Stockholm, Sweden (2018)

    Google Scholar 

  25. Corbett-Davies, S., Goel, S.: The measure and mismeasure of fairness: a critical review of fair machine learning. http://arxiv.org/abs/1808.00023 (2018)

  26. Silva, S., Kenney, M.: Algorithms, platforms, and ethnic bias. Commun. ACM. 62, 37–39 (2019)

    Article  Google Scholar 

  27. Bellamy, R.K.E., et al.: AI fairness 360: an extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. IBM J. Res. Dev. 63 (2018)

    Google Scholar 

  28. vom Brocke, J., Simons, A., Riemer, K., Niehaves, B., Plattfaut, R., Cleven, A.: Standing on the shoulders of giants: challenges and recommendations of literature search in information systems research. Commun. Assoc. Inf. Syst. 37, 205–224 (2015)

    Google Scholar 

  29. Rowe, F.: What literature review is not: diversity, boundaries and recommendations. Eur. J. Inf. Syst. 23, 241–255 (2014)

    Article  Google Scholar 

  30. Schryen, G.: Revisiting IS business value research: what we already know, what we still need to know, and how we can get there. Eur. J. Inf. Syst. 22, 139–169 (2013)

    Article  Google Scholar 

  31. Levy, Y., Ellis, T.J.: A systems approach to conduct an effective literature review in support of information systems research. Informing Sci. J. 9, 181–211 (2006)

    Article  Google Scholar 

  32. Webster, J., Watson, R.T.: Analyzing the past to prepare for the future: writing a literature review. MISQ 26, xiii–xxiii (2002)

    Google Scholar 

  33. Randolph, J.: A guide to writing the dissertation literature review. Pract. Assess. Res. Evaluation 14, 13 (2009)

    Google Scholar 

  34. Fink, A.: Conducting Research Literature Reviews: From the Internet to Paper. SAGE Publications (2019)

    Google Scholar 

  35. Viera, A.J., Garrett, J.M.: Understanding interobserver agreement: the kappa statistic. Fam. Med. 37, 360–363 (2005)

    Google Scholar 

  36. Baer, T.: Understand, Manage, and Prevent Algorithmic Bias. A Guide for Business Users and Data Scientists. Apress, Berkeley (2019)

    Google Scholar 

  37. Zarya, V.: The share of female CEOs in the fortune 500 dropped by 25% in 2018. Fortune.com (2018)

    Google Scholar 

  38. d’Alessandro, B., O’Neil, C., LaGatta, T.: Conscientious classification: a data scientist’s guide to discrimination-aware classification. Big Data. 5, 120–134 (2017)

    Article  Google Scholar 

  39. Lum, K., Isaac, W.: To predict and serve? Significance 13, 14–19 (2016)

    Article  Google Scholar 

  40. Angwin, J., Larson, J., Mattu, S., Kirchner, L.: Machine Bias. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

  41. Lan, J., Hu, M.Y., Patuwo, E., Zhang, G.P.: An investigation of neural network classifiers with unequal misclassification costs and group sizes. Decis. Support Syst. 48, 582–591 (2010)

    Article  Google Scholar 

  42. Suresh, H., Gong, J.J., Guttag, J.V.: Learning tasks for multitask learning. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, New York, USA, pp. 802–810. ACM (2018)

    Google Scholar 

  43. Chouldechova, A.: Fair prediction with disparate impact: a study of bias in recidivism prediction instruments. Big Data 5, 153–163 (2017)

    Article  Google Scholar 

  44. Collins, E.: Punishing risk. Georgetown Law J. 107, 57 (2018)

    Google Scholar 

  45. Martin, K.: Designing ethical algorithms. MIS Q. Exec. 18, 129–142 (2019)

    Article  Google Scholar 

  46. Jones, M.: What we talk about when we talk about (big) data. J. Strateg. Inf. Syst. 28, 3–16 (2019)

    Article  Google Scholar 

  47. Barocas, S., Boyd, D.: Engaging the ethics of data science in practice. Commun. ACM. 60, 23–25 (2017)

    Article  Google Scholar 

  48. Buolamwini, J., Gebru, T.: Gender shades: intersectional accuracy disparities in commercial gender classification. In: Proceedings of the 1st Conference on Fairness, Accountability and Transparency, New York, USA, pp. 77–91. PMLR (2018)

    Google Scholar 

  49. Kamiran, F., Calders, T.: Data preprocessing techniques for classification without discrimination. Knowl. Inf. Syst. 33, 1–33 (2012)

    Article  Google Scholar 

  50. Kamiran, F., Calders, T.: Classifying without discriminating. In: 2nd International Conference on Computer, Control and Communication, pp. 1–6. IEEE (2009)

    Google Scholar 

  51. Chen, I.Y., Johansson, F.D., Sontag, D.: Why is my classifier discriminatory? In: Advances in Neural Information Processing Systems 31 (NIPS 2018), pp. 3539–3550 (2018)

    Google Scholar 

  52. Hajian, S., Domingo-Ferrer, J.: A methodology for direct and indirect discrimination prevention in data mining. IEEE Trans. Knowl. Data Eng. 25, 1445–1459 (2013)

    Article  Google Scholar 

  53. Kamiran, F., Žliobaite, I., Calders, T.: Quantifying explainable discrimination and removing illegal discrimination in automated decision making. Knowl. Inf. Syst. 35, 613–644 (2013)

    Article  Google Scholar 

  54. Calmon, F., Wei, D., Vinzamuri, B., Natesan Ramamurthy, K., Varshney, K.R.: Optimized pre-processing for discrimination prevention. In: Advances in Neural Information Processing Systems 30, pp. 3992–4001. Curran Associates, Inc. (2017)

    Google Scholar 

  55. Kamishima, T., Akaho, S., Asoh, H., Sakuma, J.: Fairness-aware classifier with prejudice remover regularizer. In: Flach, P.A., De Bie, T., Cristianini, N. (eds.) ECML PKDD 2012. LNCS (LNAI), vol. 7524, pp. 35–50. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33486-3_3

    Chapter  Google Scholar 

  56. Zafar, M.B., Valera, I., Rodriguez, M.G., Gummadi, K.P.: Fairness constraints: mechanisms for fair classification. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), Fort Lauderdale, Florida, USA. PMLR (2015)

    Google Scholar 

  57. Zhang, B.H., Lemoine, B., Mitchell, M.: Mitigating unwanted biases with adversarial learning. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, New York, USA, pp. 335–340. ACM (2018)

    Google Scholar 

  58. Calders, T., Verwer, S.: Three naive Bayes approaches for discrimination-free classification. Data Min. Knowl. Discov. 21, 277–292 (2010)

    Article  Google Scholar 

  59. Binder, A., Bach, S., Montavon, G., Müller, K.R., Samek, W.: Layer-wise relevance propagation for deep neural network architectures. In: Kim, K., Joukov, N. (eds.) Information Science and Applications (ICISA) 2016. LNEE, vol. 376, pp. 913–922. Springer, Singapore (2016). https://doi.org/10.1007/978-981-10-0557-2_87

  60. Berardi, V.L., Patuwo, B.E., Hu, M.Y.: A principled approach for building and evaluating neural network classification models. Decis. Support Syst. 38, 233–246 (2004)

    Article  Google Scholar 

  61. Hardt, M., Price, E., Srebro, N.: Equality of opportunity in supervised learning. In: Advances in Neural Information Processing Systems, pp. 3315–3323. Neural Information Processing Systems (NIPS) (2016)

    Google Scholar 

  62. Dwork, C., Immorlica, N., Kalai, A.T., Leiserson, M.: Decoupled classifiers for fair and efficient machine learning. http://arxiv.org/abs/1707.06613 (2017)

  63. Ryu, H.J., Adam, H., Mitchell, M.: InclusiveFaceNet: improving face attribute detection with race and gender diversity. http://arxiv.org/abs/1712.00193 (2017)

  64. Samek, W., Wiegand, T., Müller, K.-R.: Explainable artificial intelligence: understanding, visualizing and interpreting deep learning models. http://arxiv.org/abs/1708.08296 (2017)

Download references

Author information

Authors and Affiliations

  1. Institute of Information Management, University of St. Gallen, St. Gallen, Switzerland

    Tobias Fahse, Viktoria Huber & Benjamin van Giffen

Authors
  1. Tobias Fahse
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Viktoria Huber
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Benjamin van Giffen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tobias Fahse .

Editor information

Editors and Affiliations

  1. Faculty of Business Administration and Economics, University of Duisburg-Essen, Essen, Germany

    Frederik Ahlemann

  2. Faculty of Business Administration and Economics, University of Duisburg-Essen, Essen, Germany

    Reinhard Schütte

  3. Department of Computer Science and Applied Cognitive Science, University of Duisburg-Essen, Duisburg, Germany

    Stefan Stieglitz

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fahse, T., Huber, V., van Giffen, B. (2021). Managing Bias in Machine Learning Projects. In: Ahlemann, F., Schütte, R., Stieglitz, S. (eds) Innovation Through Information Systems. WI 2021. Lecture Notes in Information Systems and Organisation, vol 47. Springer, Cham. https://doi.org/10.1007/978-3-030-86797-3_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-86797-3_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-86796-6

  • Online ISBN: 978-3-030-86797-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation