Abstract
Published studies have suggested the bias of automated face-based gender classification algorithms across gender-race groups. Specifically, unequal accuracy rates were obtained for women and dark-skinned people. To mitigate the bias of gender classifiers, the vision community has developed several strategies. However, the efficacy of these mitigation strategies is demonstrated for a limited number of races mostly, Caucasian and African-American. Further, these strategies often offer a trade-off between bias and classification accuracy. To further advance the state-of-the-art, we leverage the power of generative views, structured learning, and evidential learning towards mitigating gender classification bias. We demonstrate the superiority of our bias mitigation strategy in improving classification accuracy and reducing bias across gender-racial groups through extensive experimental validation, resulting in state-of-the-art performance in intra- and cross dataset evaluations.
Supported by National Science Foundation (NSF).
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Studies in [25, 49] have shown the inherent problems with gender and race classification. While the datasets used in this study use an almost balanced dataset for the training, it still lacks representation of entire large demographic groups, effectively erasing such categories. We compare the gender and racial categories specified in these datasets to past studies, not to reaffirm or encourage the usage of such reductive classifications.
References
Abdal, R., Qin, Y., Wonka, P.: Image2StyleGAN: how to embed images into the StyleGAN latent space? In: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, South Korea, 27 October–2 November 2019, pp. 4431–4440. IEEE (2019). https://doi.org/10.1109/ICCV.2019.00453
Adeli-Mosabbeb, E., et al.: Representation learning with statistical independence to mitigate bias. In: IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, 3–8 January 2021, pp. 2512–2522. IEEE (2021). https://doi.org/10.1109/WACV48630.2021.00256
Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019. OpenReview.net (2019). https://openreview.net/forum?id=B1xsqj09Fm
Bui, T.D., Ravi, S., Ramavajjala, V.: Neural graph learning: training neural networks using graphs. In: Proceedings of the 11th ACM International Conference on Web Search and Data Mining, WSDM 2018, pp. 64–71. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3159652.3159731
Buolamwini, J., Gebru, T.: Gender shades: intersectional accuracy disparities in commercial gender classification. In: Friedler, S.A., Wilson, C. (eds.) Conference on Fairness, Accountability and Transparency, FAT 2018. Proceedings of Machine Learning Research, New York, NY, USA, 23–24 February 2018, vol. 81, pp. 77–91. PMLR (2018). http://proceedings.mlr.press/v81/buolamwini18a.html
Chai, L., Zhu, J., Shechtman, E., Isola, P., Zhang, R.: Ensembling with deep generative views. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, 19–25 June 2021, pp. 14997–15007. Computer Vision Foundation/IEEE (2021). https://openaccess.thecvf.com/content/CVPR2021/html/Chai_Ensembling_With_Deep_Generative_Views_CVPR_2021_paper.html
Chuang, C., Mroueh, Y.: Fair mixup: fairness via interpolation. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, 3–7 May 2021. OpenReview.net (2021). https://openreview.net/forum?id=DNl5s5BXeBn
Collins, E., Bala, R., Price, B., Süsstrunk, S.: Editing in style: uncovering the local semantics of GANs. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13–19 June 2020, pp. 5770–5779. Computer Vision Foundation/IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00581
Creswell, A., Bharath, A.A.: Inverting the generator of a generative adversarial network. IEEE Trans. Neural Netw. Learn. Syst. 30(7), 1967–1974 (2019). https://doi.org/10.1109/TNNLS.2018.2875194
Das, A., Dantcheva, A., Bremond, F.: Mitigating bias in gender, age and ethnicity classification: a multi-task convolution neural network approach. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11129, pp. 573–585. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11009-3_35
Denton, E., Hutchinson, B., Mitchell, M., Gebru, T.: Detecting bias with generative counterfactual face attribute augmentation. CoRR abs/1906.06439 (2019). http://arxiv.org/abs/1906.06439
Goetschalckx, L., Andonian, A., Oliva, A., Isola, P.: GANalyze: toward visual definitions of cognitive image properties. In: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, South Korea, 27 October–2 November 2019, pp. 5743–5752. IEEE (2019). https://doi.org/10.1109/ICCV.2019.00584
Gong, S., Liu, X., Jain, A.K.: DebFace: de-biasing face recognition. CoRR abs/1911.08080 (2019). http://arxiv.org/abs/1911.08080
Goodfellow, I., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 27. Curran Associates, Inc. (2014). https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings, San Diego, CA, USA, 7–9 May 2015 (2015). http://arxiv.org/abs/1412.6572
Guan, S., Tai, Y., Ni, B., Zhu, F., Huang, F., Yang, X.: Collaborative learning for faster StyleGAN embedding. CoRR abs/2007.01758 (2020). https://arxiv.org/abs/2007.01758
Härkönen, E., Hertzmann, A., Lehtinen, J., Paris, S.: GANSpace: discovering interpretable GAN controls. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS, Virtual, December 2020, pp. 6–12 (2020). https://proceedings.neurips.cc/paper/2020/hash/6fe43269967adbb64ec6149852b5cc3e-Abstract.html
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017). https://proceedings.neurips.cc/paper/2017/file/8a1d694707eb0fefe65871369074926d-Paper.pdf
Jain, A., Huang, J.: Integrating independent components and linear discriminant analysis for gender classification. In: 6th IEEE International Conference on Automatic Face and Gesture Recognition, FGR 2004, 17–19 May 2004, Seoul, Korea, pp. 159–163. IEEE Computer Society (2004). https://doi.org/10.1109/AFGR.2004.1301524
Kamiran, F., Karim, A., Zhang, X.: Decision theory for discrimination-aware classification. In: 2012 IEEE 12th International Conference on Data Mining, pp. 924–929 (2012). https://doi.org/10.1109/ICDM.2012.45
Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: 6th International Conference on Learning Representations, ICLR 2018, Conference Track Proceedings, Vancouver, BC, Canada, 30 April–3 May 2018. OpenReview.net (2018). https://openreview.net/forum?id=Hk99zCeAb
Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., Aila, T.: Training generative adversarial networks with limited data. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, Virtual, December 2020, pp. 6–12 (2020). https://proceedings.neurips.cc/paper/2020/hash/8d30aa96e72440759f74bd2306c1fa3d-Abstract.html
Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 4401–4410. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPR.2019.00453
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13–19 June 2020, pp. 8107–8116. Computer Vision Foundation/IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00813
Keyes, O.: The misgendering machines: Trans/HCI implications of automatic gender recognition. Proc. ACM Hum. Comput. Interact. 2(CSCW), 88:1–88:22 (2018). https://doi.org/10.1145/3274357
Khan, S., Ahmad, M., Nazir, M., Riaz, N.: A comparative analysis of gender classification techniques. Middle-East J. Sci. Res. 20, 1–13 (2014). https://doi.org/10.5829/idosi.mejsr.2014.20.01.11434
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: 5th International Conference on Learning Representations, ICLR 2017, Conference Track Proceedings, Toulon, France, 24–26 April 2017. OpenReview.net (2017). https://openreview.net/forum?id=SJU4ayYgl
Krishnan, A., Almadan, A., Rattani, A.: Understanding fairness of gender classification algorithms across gender-race groups. In: Wani, M.A., Luo, F., Li, X.A., Dou, D., Bonchi, F. (eds.) 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020, Miami, FL, USA, 14–17 December 2020, pp. 1028–1035. IEEE (2020). https://doi.org/10.1109/ICMLA51294.2020.00167
Krishnan, A., Almadan, A., Rattani, A.: Investigating fairness of ocular biometrics among young, middle-aged, and older adults. In: 2021 International Carnahan Conference on Security Technology (ICCST), pp. 1–7 (2021). https://doi.org/10.1109/ICCST49569.2021.9717383
Krishnan, A., Almadan, A., Rattani, A.: Probing fairness of mobile ocular biometrics methods across gender on VISOB 2.0 dataset. In: Del Bimbo, A., et al. (eds.) ICPR 2021. LNCS, vol. 12668, pp. 229–243. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-68793-9_16
Kärkkäinen, K., Joo, J.: FairFace: face attribute dataset for balanced race, gender, and age (2019)
Lin, F., Wu, Y., Zhuang, Y., Long, X., Xu, W.: Human gender classification: a review. Int. J. Biom. 8(3/4), 275–300 (2016). https://doi.org/10.1504/IJBM.2016.10003589
Lipton, Z.C., Tripathi, S.: Precise recovery of latent vectors from generative adversarial networks. In: 5th International Conference on Learning Representations, ICLR 2017, Workshop Track Proceedings, Toulon, France, 24–26 April 2017. OpenReview.net (2017). https://openreview.net/forum?id=HJC88BzFl
Mahajan, D., et al.: Exploring the limits of weakly supervised pretraining. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 185–201. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_12
Majumdar, P., Singh, R., Vatsa, M.: Attention aware debiasing for unbiased model prediction. In: IEEE/CVF International Conference on Computer Vision Workshops, ICCVW 2021, Montreal, BC, Canada, 11–17 October 2021, pp. 4116–4124. IEEE (2021). https://doi.org/10.1109/ICCVW54120.2021.00459
Miyato, T., Maeda, S., Koyama, M., Ishii, S.: Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell. (2017). https://doi.org/10.1109/TPAMI.2018.2858821
Morales, A., Fiérrez, J., Vera-Rodríguez, R., Tolosana, R.: SensitiveNets: learning agnostic representations with application to face images. IEEE Trans. Pattern Anal. Mach. Intell. 43(6), 2158–2164 (2021). https://doi.org/10.1109/TPAMI.2020.3015420, https://doi.org/10.1109/TPAMI.2020.3015420
Muthukumar, V.: Color-theoretic experiments to understand unequal gender classification accuracy from face images. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 2286–2295. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPRW.2019.00282
Nadimpalli, A.V., Rattani, A.: GBDF: gender balanced DeepFake dataset towards fair DeepFake detection (2022). https://doi.org/10.48550/ARXIV.2207.10246. https://arxiv.org/abs/2207.10246
Nagpal, S., Singh, M., Singh, R., Vatsa, M.: Attribute aware filter-drop for bias-invariant classification. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 147–153 (2020). https://doi.org/10.1109/CVPRW50498.2020.00024
Ngan, M., Grother, P.: Face recognition vendor test (FRVT) - performance of automated gender classification algorithms (2015). https://doi.org/10.6028/NIST.IR.8052
Nguyen, H.M., Reddy, N., Rattani, A., Derakhshani, R.: VISOB 2.0 - the second international competition on mobile ocular biometric recognition. In: Del Bimbo, A., et al. (eds.) ICPR 2021. LNCS, vol. 12668, pp. 200–208. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-68793-9_14
Park, S., Kim, D., Hwang, S., Byun, H.: README: representation learning by fairness-aware disentangling method. CoRR abs/2007.03775 (2020). https://arxiv.org/abs/2007.03775
Perarnau, G., van de Weijer, J., Raducanu, B.C., Álvarez, J.M.: Invertible conditional GANs for image editing. CoRR abs/1611.06355 (2016). http://arxiv.org/abs/1611.06355
Radford, A., et al.: Learning transferable visual models from natural language supervision. CoRR abs/2103.00020 (2021). https://arxiv.org/abs/2103.00020
Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: Bengio, Y., LeCun, Y. (eds.) 4th International Conference on Learning Representations, ICLR 2016, Conference Track Proceedings, San Juan, Puerto Rico, 2–4 May 2016 (2016). http://arxiv.org/abs/1511.06434
Raji, I.D., Buolamwini, J.: Actionable auditing: investigating the impact of publicly naming biased performance results of commercial AI products. In: Conitzer, V., Hadfield, G.K., Vallor, S. (eds.) Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES 2019, Honolulu, HI, USA, 27–28 January 2019, pp. 429–435. ACM (2019). https://doi.org/10.1145/3306618.3314244
Ramaswamy, V.V., Kim, S.S.Y., Russakovsky, O.: Fair attribute classification through latent space de-biasing. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, Virtual, 19–25 June 2021, pp. 9301–9310. Computer Vision Foundation/IEEE (2021). https://openaccess.thecvf.com/content/CVPR2021/html/Ramaswamy_Fair_Attribute_Classification_Through_Latent_Space_De-Biasing_CVPR_2021_paper.html
Randall, D.W.: Geoffrey Bowker and Susan Leigh Star, sorting things out: classification and its consequences - review. Comput. Support. Coop. Work. 10(1), 147–153 (2001). https://doi.org/10.1023/A:1011229919958
Ricanek, K., Tesafaye, T.: MORPH: a longitudinal image database of normal adult age-progression. In: 7th International Conference on Automatic Face and Gesture Recognition (FGR06), pp. 341–345 (2006). https://doi.org/10.1109/FGR.2006.78
Richardson, E., et al.: Encoding in style: a StyleGAN encoder for image-to-image translation. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, Virtual, 19–25 June 2021, pp. 2287–2296. Computer Vision Foundation/IEEE (2021). https://openaccess.thecvf.com/content/CVPR2021/html/Richardson_Encoding_in_Style_A_StyleGAN_Encoder_for_Image-to-Image_Translation_CVPR_2021_paper.html
Ryu, H.J., Adam, H., Mitchell, M.: InclusiveFaceNet: improving face attribute detection with race and gender diversity. In: Workshop on Fairness, Accountability, and Transparency in Machine Learning, pp. 1–6, March 2018
Sensoy, M., Kaplan, L.M., Kandemir, M.: Evidential deep learning to quantify classification uncertainty. In: Bengio, S., Wallach, H.M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montréal, Canada, December 2018, pp. 3–8 (2018). https://proceedings.neurips.cc/paper/2018/hash/a981f2b708044d6fb4a71a1463242520-Abstract.html
Shen, Y., Gu, J., Tang, X., Zhou, B.: Interpreting the latent space of GANs for semantic face editing. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13–19 June 2020, pp. 9240–9249. Computer Vision Foundation/IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00926
Shen, Y., Zhou, B.: Closed-form factorization of latent semantics in GANs. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, Virtual, 19–25 June 2021, pp. 1532–1540. Computer Vision Foundation/IEEE (2021). https://openaccess.thecvf.com/content/CVPR2021/html/Shen_Closed-Form_Factorization_of_Latent_Semantics_in_GANs_CVPR_2021_paper.html
Siddiqui, H., Rattani, A., Ricanek, K., Hill, T.: An examination of bias of facial analysis based BMI prediction models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2926–2935, June 2022
Singh, R., Majumdar, P., Mittal, S., Vatsa, M.: Anatomizing bias in facial analysis. CoRR abs/2112.06522 (2021). https://arxiv.org/abs/2112.06522
Smith, P., Chen, C.: Transfer learning with deep CNNs for gender recognition and age estimation (2018)
Tartaglione, E., Barbano, C.A., Grangetto, M.: EnD: entangling and disentangling deep representations for bias correction. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, 19–25 June 2021, pp. 13508–13517. Computer Vision Foundation/IEEE (2021)
TensorFlow: the neural structured learning framework. https://www.tensorflow.org/neural_structured_learning/framework
Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., Cohen-Or, D.: Designing an encoder for StyleGAN image manipulation. ACM Trans. Graph. 40(4), 133:1–133:14 (2021). https://doi.org/10.1145/3450626.3459838
Verma, S., Rubin, J.: Fairness definitions explained. In: Proceedings of the International Workshop on Software Fairness, FairWare 2018, pp. 1–7. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3194770.3194776
Voynov, A., Babenko, A.: Unsupervised discovery of interpretable directions in the GAN latent space. In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13–18 July 2020, Virtual Event. Proceedings of Machine Learning Research, vol. 119, pp. 9786–9796. PMLR (2020). http://proceedings.mlr.press/v119/voynov20a.html
Wang, B., Ponce, C.R.: A geometric analysis of deep generative image models and its applications. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, 3–7 May 2021. OpenReview.net (2021). https://openreview.net/forum?id=GH7QRzUDdXG
Wang, T., Zhao, J., Yatskar, M., Chang, K., Ordonez, V.: Balanced datasets are not enough: estimating and mitigating gender bias in deep image representations. In: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, South Korea, 27 October–2 November 2019, pp. 5309–5318. IEEE (2019). https://doi.org/10.1109/ICCV.2019.00541
Wayman, J.: Large-scale civilian biometric systems-issues and feasibility. In: Proceedings of Card Tech/Secur Tech ID, vol. 732 (1997)
Wikipedia: Jensen-Shannon divergence, May 2022. https://en.wikipedia.org/wiki/Jensen-Shannon_divergence
Wu, Z., Lischinski, D., Shechtman, E.: StyleSpace analysis: disentangled controls for StyleGAN image generation. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, Virtual, 19–25 June 2021, pp. 12863–12872. Computer Vision Foundation/IEEE (2021)
Zanlorensi, L.A., Laroca, R., Lucio, D.R., Santos, L.R., Britto, A.S., Menotti, D.: UFPR-periocular: a periocular dataset collected by mobile devices in unconstrained scenarios (2020). https://doi.org/10.48550/ARXIV.2011.12427. https://arxiv.org/abs/2011.12427
Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multi-task cascaded convolutional networks. CoRR abs/1604.02878 (2016). http://arxiv.org/abs/1604.02878
Zhang, Z., Song, Y., Qi, H.: Age progression/regression by conditional adversarial autoencoder. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2017)
Acknowledgement
This work is supported from National Science Foundation (NSF) award no. 2129173. The research infrastructure used in this study is supported in part from a grant no. 13106715 from the Defense University Research Instrumentation Program (DURIP) from Air Force Office of Scientific Research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 Springer Nature Switzerland AG
About this paper
Cite this paper
Ramachandran, S., Rattani, A. (2023). Deep Generative Views to Mitigate Gender Classification Bias Across Gender-Race Groups. In: Rousseau, JJ., Kapralos, B. (eds) Pattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges. ICPR 2022. Lecture Notes in Computer Science, vol 13645. Springer, Cham. https://doi.org/10.1007/978-3-031-37731-0_40
Download citation
DOI: https://doi.org/10.1007/978-3-031-37731-0_40
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-37730-3
Online ISBN: 978-3-031-37731-0
eBook Packages: Computer ScienceComputer Science (R0)