Abstract
Given the increasing prevalence of facial analysis technology, the problem of bias in these tools is becoming an even greater source of concern. Causality has been proposed as a method to address the problem of bias, giving rise to the popularity of using counterfactuals as a bias mitigation tool. In this paper, we undertake a systematic investigation of the usage of counterfactuals to achieve both statistical and causal-based fairness in facial expression recognition. We explore bias mitigation strategies with counterfactual data augmentation at the pre-processing, in-processing, and post-processing stages as well as a stacked approach that combines all three methods. At the in-processing stage, we propose using Siamese Networks to suppress the differences between the predictions on the original and the counterfactual images. Our experimental results on RAF-DB with counterfactuals added show that: (1) The in-processing method outperforms at the pre-processing and post-processing stages, in terms of accuracy, F1 score, statistical fairness and counterfactual fairness, and (2) stacking the pre-processing, in-processing and post-processing stages provides the best performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Barocas, S., Hardt, M., Narayanan, A.: Fairness in machine learning. NIPS Tutor. 1, 2 (2017)
Binns, R.: Fairness in machine learning: Lessons from political philosophy. In: Conference on Fairness, Accountability and Transparency (2018)
Bromley, J., et al.: Signature verification using a “Siamese’’ time delay neural network. Int. J. Pattern Recogn. Artif. Intell. 7(04), 669–688 (1993)
Buolamwini, J., Gebru, T.: Gender shades: intersectional accuracy disparities in commercial gender classification. In: Conference on Fairness, Accountability and Transparency, pp. 77–91. PMLR (2018)
Cheong, J., Kalkan, S., Gunes, H.: The hitchhiker’s guide to bias and fairness in facial affective signal processing: overview and techniques. IEEE Signal Process. Mag. 38(6), 39–49 (2021)
Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: CVPR (2005)
Chouldechova, A.: Fair prediction with disparate impact: a study of bias in recidivism prediction instruments. Big Data 5(2), 153–163 (2017)
Churamani, N., Kara, O., Gunes, H.: Domain-incremental continual learning for mitigating bias in facial expression and action unit recognition. arXiv preprint arXiv:2103.08637 (2021)
Crawford, K.: Time to regulate AI that interprets human emotions. Nature 592(7853), 167–167 (2021)
Dash, S., Balasubramanian, V.N., Sharma, A.: Evaluating and mitigating bias in image classifiers: a causal perspective using counterfactuals. In: WACV (2022)
Davani, A.M., Omrani, A., Kennedy, B., Atari, M., Ren, X., Dehghani, M.: Fair hate speech detection through evaluation of social group counterfactuals. arXiv preprint arXiv:2010.12779 (2020)
Davani, A.M., Omrani, A., Kennedy, B., Atari, M., Ren, X., Dehghani, M.: Improving counterfactual generation for fair hate speech detection. In: Workshop on Online Abuse and Harms (WOAH) (2021)
Denton, E., Hutchinson, B., Mitchell, M., Gebru, T.: Detecting bias with generative counterfactual face attribute augmentation. arXiv e-prints, pp. arXiv-1906 (2019)
Dinan, E., Fan, A., Williams, A., Urbanek, J., Kiela, D., Weston, J.: Queens are powerful too: mitigating gender bias in dialogue generation. In: EMNLP (2020)
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pp. 214–226 (2012)
Ekman, R.: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, USA (1997)
Friedler, S.A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E.P., Roth, D.: A comparative study of fairness-enhancing interventions in machine learning. In: Conference on Fairness, Accountability, and Transparency (2019)
Gajane, P., Pechenizkiy, M.: On formalizing fairness in prediction with machine learning. arXiv preprint arXiv:1710.03184 (2017)
Garcia, R., Wandzik, L., Grabner, L., Krueger, J.: The harms of demographic bias in deep face recognition research. In: Proceedings of International Conference on Biometrics (ICB), pp. 1–6 (2019)
Gunes, H., Schuller, B.: Categorical and dimensional affect analysis in continuous input: current trends and future directions. Image Vis. Comput. 31(2), 120–136 (2013)
Hanna, A., Denton, E., Smart, A., Smith-Loud, J.: Towards a critical race methodology in algorithmic fairness. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 501–512 (2020)
Hardt, M., Price, E., Srebro, N.: Equality of opportunity in supervised learning. In: NIPS (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
Hoffman, A.: Where fairness fails: data, algorithms and the limits of antidiscrimination discourse. J. Inf. Commun. Soc. 22, 900–915 (2019)
Howard, A., Zhang, C., Horvitz, E.: Addressing bias in machine learning algorithms: A pilot study on emotion recognition for intelligent systems. In: Proceedings of Advanced Robotics Social Impacts (ARSO) (2017)
Jain, N., Olmo, A., Sengupta, S., Manikonda, L., Kambhampati, S.: Imperfect imaganation: Implications of GANs exacerbating biases on facial data augmentation and snapchat face lenses. Artif. Intell. 304, 103652 (2022)
Joo, J., Kärkkäinen, K.: Gender slopes: Counterfactual fairness for computer vision models by attribute manipulation. In: Workshop on Fairness, Accountability, Transparency and Ethics in Multimedia (2020)
Kamiran, F., Karim, A., Zhang, X.: Decision theory for discrimination-aware classification. In: International Conference on Data Mining (2012)
Kilbertus, N., Rojas-Carulla, M., Parascandolo, G., Hardt, M., Janzing, D., Schölkopf, B.: Avoiding discrimination through causal reasoning. In: NIPS, pp. 656–666 (2017)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Kusner, M., Loftus, J., Russell, C., Silva, R.: Counterfactual fairness. In: NIPS (2017)
Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: CVPR (2017)
Liu, M., et al.: StGAN: a unified selective transfer network for arbitrary image attribute editing. In: CVPR (2019)
Loftus, J.R., Russell, C., Kusner, M.J., Silva, R.: Causal reasoning for algorithmic fairness. arXiv preprint arXiv:1805.05859 (2018)
Lu, K., Mardziel, P., Wu, F., Amancharla, P., Datta, A.: Gender bias in neural natural language processing. In: Nigam, V., et al. (eds.) Logic, Language, and Security. LNCS, vol. 12300, pp. 189–202. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-62077-6_14
Maudslay, R.H., Gonen, H., Cotterell, R., Teufel, S.: It’s all in the name: mitigating gender bias with name-based counterfactual data substitution. In: EMNLP-IJCNLP (2019)
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Comput. Surv. (CSUR) 54(6), 1–35 (2019)
Mehrabi, N., Naveed, M., Morstatter, F., Galstyan, A.: Exacerbating algorithmic bias through fairness attacks. In: AAAI (2021)
Nabi, R., Shpitser, I.: Fair inference on outcomes. In: AAAI (2018)
Ngxande, M., Tapamo, J., Burke, M.: Bias remediation in driver drowsiness detection systems using generative adversarial networks. IEEE Access 8, 55592–55601 (2020). https://doi.org/10.1109/ACCESS.2020.2981912
Niu, Y., Tang, K., Zhang, H., Lu, Z., Hua, X.S., Wen, J.R.: Counterfactual VQA: a cause-effect look at language bias. In: CVPR (2021)
Pearl, J.: Causality. Cambridge University Press, Cambridge (2009)
Russell, J.A.: A circumplex model of affect. J. Pers. Soc. Psychol. 39(6), 1161 (1980)
Salimi, B., Rodriguez, L., Howe, B., Suciu, D.: Interventional fairness: causal database repair for algorithmic fairness. In: International Conference on Management of Data (2019)
Sariyanidi, E., Gunes, H., Cavallaro, A.: Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE TPAMI 37(6), 1113–1133 (2014)
Verma, S., Rubin, J.: Fairness definitions explained. In: International Workshop on Software Fairness (Fairware), pp. 1–7. IEEE (2018)
Wang, W., Feng, F., He, X., Zhang, H., Chua, T.S.: Clicks can be cheating: counterfactual recommendation for mitigating clickbait issue. In: ACM SIGIR Conference on Research and Development in Information Retrieval (2021)
Wong, A.: Mitigating gender bias in neural machine translation using counterfactual data. M.A. thesis, City University of New York (2020)
Xu, T., White, J., Kalkan, S., Gunes, H.: Investigating bias and fairness in facial expression recognition. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12540, pp. 506–523. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-65414-6_35
Zafar, M.B., Valera, I., Rogriguez, M.G., Gummadi, K.P.: Fairness constraints: mechanisms for fair classification. In: Artificial Intelligence and Statistics, pp. 962–970. PMLR (2017)
Acknowledgement
J. Cheong is supported by the Alan Turing Institute doctoral studentship and the Cambridge Commonwealth Trust. H. Gunes’ work is supported by the EPSRC under grant ref. EP/R030782/1.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Cheong, J., Kalkan, S., Gunes, H. (2023). Counterfactual Fairness for Facial Expression Recognition. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13805. Springer, Cham. https://doi.org/10.1007/978-3-031-25072-9_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-25072-9_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25071-2
Online ISBN: 978-3-031-25072-9
eBook Packages: Computer ScienceComputer Science (R0)