Skip to main content
SpringerLink
Log in

Deception Detection on “Bag-of-Lies”: Integration of Multi-modal Data Using Machine Learning Algorithms

  • Conference paper
  • First Online:
Proceedings of International Conference on Machine Intelligence and Data Science Applications

Part of the book series: Algorithms for Intelligent Systems ((AIS))

Abstract

In recent years, deception detection has shown an increasing amount of attention due to the considerable growth in digital media. It is a prevalent issue in security and ethical concerns. Recent methods to detect deception, such as the polygraph test were mainly focused on applications of law enforcement, which had proven to falsely accuse the innocent and free the guilty in multiple cases. Further, deception detection has been widely studied using traditional modalities, such as video, audio, and transcripts. In this study, a combination of contact and non-contact based method is proposed, which considers data of three different modalities, namely, video, audio, and electroencephalogram (EEG) signals. First, 20 frames from each video are fetched and cropped the facial portion to focus on facial movements and concatenated to form a single image. Moreover, an audio signal is detached from video and is plotted into two dimensional (2D) plane to form an image. Furthermore, 13 channels of EEG signals are plotted into 2D-plane and concatenated to generate an image. The proposed method is evaluated on the “Bag-of-lies”, and Real Life (RL) datasets using known classification algorithms viz., k-nearest neighbors, support vector machine, random forest, multi-layer perceptron, and adaBoost. Our experimental results indicate that the highest average accuracy with 70% on “Bag-of-lies” dataset is achieved when three scores obtained on three modalities are combined, and “RL” dataset is achieved 76% when two modalities are combined, such as video and audio, which is better than the existing method. ...

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 259.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 329.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 329.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Dalmaijer ES, Mathôt S, Van der Stigchel S (2014) Pygaze: an open-source, cross-platform toolbox for minimal-effort programming of eyetracking experiments. Behav Res Methods 46(4):913–921

    Article  Google Scholar 

  2. Saxe L (1991) Science and the cqt polygraph. Integr Physiol Behav Sci 26(3):223–231

    Article  MathSciNet  Google Scholar 

  3. Cui Q, Vanman EJ, Wei D, Yang W, Jia L, Zhang Q (2013) Detection of deception based on fmri activation patterns underlying the production of a deceptive response and receiving feedback about the success of the deception after a mock murder crime. Soc Cognit Affect Neurosci 9(10):1472–1480

    Article  Google Scholar 

  4. Vrij A (2000) Detecting lies and deceit: the psychology of lying and implications for professional practice. Wiley, New York

    Google Scholar 

  5. Buller DB, Burgoon JK, White CH, Ebesu AS (1994) Interpersonal deception vii: Behavioral profiles of falsification, equivocation, and concealment. J Lang Soc Psychol 13(4):366–395

    Article  Google Scholar 

  6. Ekman P (1988) Lying and nonverbal behavior: theoretical issues and new & #xC;findings. J Nonverbal Behav 12(3):163–175

    Article  Google Scholar 

  7. Ganis G, Peter Rosenfeld J, Meixner J, Kievit RA, Schendan HE (2011) Lying in the scanner: covert countermeasures disrupt deception detection by functional magnetic resonance imaging. Neuroimage 55(1):312–319

    Article  Google Scholar 

  8. Andrew Kozel F, Johnson KA, Mu Q, Grenesko EL, Laken SJ, George MS (2005) Detecting deception using functional magnetic resonance imaging. Biol Psychiatry 58(8):605–613

    Article  Google Scholar 

  9. Langleben DD, Campbell Moriarty J (2013) Using brain imaging for lie detection: where science, law, and policy collide. Psychol Public Policy Law 19(2):222

    Article  Google Scholar 

  10. DePaulo BM, Lindsay JJ, Malone BE, Muhlenbruck L, Charlton K, Cooper H (2003) Cues to deception. Psychol Bull 129(1):74

    Article  Google Scholar 

  11. Zhang Z, Singh V, Slowe TE, Tulyakov S, Govindaraju V (2007) Real-time automatic deceit detection from involuntary facial expressions. In: 2007 IEEE conference on computer vision and pattern recognition. IEEE, pages 1–6

    Google Scholar 

  12. Pavlidis I, Eberhardt NL, Levine JA (2002) Human behaviour: seeing through the face of deception. Nature 415(6867):35

    Article  Google Scholar 

  13. Buddharaju P, Pavlidis IT, Tsiamyrtzis P (2005) Physiology-based face recognition. In: IEEE conference on advanced video and signal based surveillance. IEEE, pp 354–359

    Google Scholar 

  14. Lu Y, Cederbaum AI (2005) Cisplatin-induced hepatotoxicity is enhanced by elevated expression of cytochrome p450 2e1. Toxicol Sci 89(2):515–523

    Article  Google Scholar 

  15. Meservy TO, Jensen ML, Kruse J, Burgoon JK, Nunamaker JF, Twitchell DP, Tsechpenakis G, Metaxas DM (2005) Deception detection through automatic, unobtrusive analysis of nonverbal behavior. IEEE Intell Syst 20(5):36–43

    Article  Google Scholar 

  16. Burgoon JK, Twitchell DP, Jensen ML, Meservy TO, Adkins M, Kruse J, Deokar AV, Tsechpenakis G, Lu S, Metaxas DN et al (2009) Detecting concealment of intent in transportation screening: a proof of concept. IEEE Trans Intell Transp Syst 10(1):103–112

    Article  Google Scholar 

  17. Jensen ML, Thomas O Meservy, Judee K Burgoon, and Jay F Nunamaker. Au- tomatic, multimodal evaluation of human interaction. Group Decis Negot 19(4):367–389

    Google Scholar 

  18. Nunamaker JF, Burgoon JK, Twyman NW, Gainer Proudfoot J, Schuetzler R, Scott Giboney J (2012) Establishing a foundation for automated human credibility screening. In: 2012 IEEE international conference on intelligence and security informatics. IEEE, pp 202–211

    Google Scholar 

  19. Zhang H, Cheng P, Wu J, Shi L, Chen J (2014) Online deception attack against remote state estimation. IFAC Proc 47(3):128–133

    Article  Google Scholar 

  20. Ekman P, Friesen WV (1977) Nonverbal behavior. Communication and social interaction, pages 37–46

    Google Scholar 

  21. Abouelenien M, Perez-Rosas V, Mihalcea R, Burzo M (2014). De ception detection using a multimodal approach. In: Proceedings of the 16th international conference on multimodal interaction, pages 58–65

    Google Scholar 

  22. Perez-Rosas V, Abouelenien M, Mihalcea R, Xiao Y, Linton CJ, Burzo M (2015) Verbal and nonverbal clues for real-life deception detection. In: Proceedings of the 2015 conference on empirical methods in natural language processing, pp 2336–2346

    Google Scholar 

  23. Abouelenien M, Burzo M, Mihalcea R (2016) Human acute stress detection via integration of physiological signals and thermal imaging. In: Proceedings of the 9th ACM international conference on pervasive technologies related to assistive environments, pp 1–8

    Google Scholar 

  24. Gupta V, Agarwal M, Arora M, Chakraborty T, Singh R, Vatsa M (2019) Bag-of-lies: a multimodal dataset for deception detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops

    Google Scholar 

  25. Charles F, Jr Bond, DePaulo BM (2006) Accuracy of deception judgments. Person Soc Psychol Rev 10(3):214–234

    Google Scholar 

  26. Abootalebi V, Moradi MH, Khalilzadeh MA (2009) A new approach for eeg feature extraction in p300-based lie detection. Comput Methods Prog Biomed 94(1):48–57

    Article  Google Scholar 

  27. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  Google Scholar 

  28. Sharma KK, Seal A (2019) Modeling uncertain data using monte carlo integration method for clustering. Expert Syst Appl 137:100–116

    Article  Google Scholar 

  29. Bhattacharjee D, Seal A, Ganguly S, Nasipuri M, Basu DK (2012) A comparative study of human thermal face recognition based on haar wavelet transform and local binary pattern. Comput Intell Neurosci

    Google Scholar 

  30. Inunganbi S, Seal A, Khanna P (2018) Classification of food images through interactive image segmentation. In: Asian conference on intelligent information and database systems. Springer, pp 519–528

    Google Scholar 

  31. Cortes C, Vapnik V (1995) Support vector machine. Mach Learn 20(3):273–297

    MATH  Google Scholar 

  32. Seal A, Bhattacharjee D, Nasipuri M, Rodrguez-Esparragon D, Menasalvas E, Gonzalo-Martin C (2018) PET-CT image fusion using random forest and a-trous wavelet transform. Int J Numer Methods Biomed Eng 34(3):e2933

    Article  MathSciNet  Google Scholar 

  33. Seal A, Bhattacharjee D, Nasipuri M (2016) Human face recognition using random forest based fusion of a-trous wavelet transform coefficients from thermal and visible images. AEU-Int J Electron Commun 70(8):1041–1049

    Article  Google Scholar 

  34. Seal A, Garcia-Pedrero A, Bhattacharjee D, Nasipuri M, Lillo-Saavedra M, Menasalvas E, Gonzalo-Martin C (2019) Multi-scale rois selection for classifying multi-spectral images. Multidimensional Syst Sig Process, pp 1–25

    Google Scholar 

  35. Seal A, Bhattacharjee D, Nasipuri M, Gonzalo-Martin C, Menasalvas E (2018) A-trous wavelet transform-based hybrid image fusion for face recognition using region classifiers. Expert Syst 35(6):e12307

    Article  Google Scholar 

  36. Hinton GE (1990) Connectionist learning procedures. In: Machine learning. Elsevier, pages 555–610

    Google Scholar 

  37. Van Der Malsburg C (1986) Frank rosenblatt: principles of neurodynamics: perceptrons and the theory of brain mechanisms. In: Brain theory. Springer, pages 245–248

    Google Scholar 

  38. Seal A, Bhattacharjee D, Nasipuri M, Basu DK (2011) Minu- tiae based thermal face recognition using blood perfusion data. In 2011 International Conference on Image Information Processing. IEEE, pages 1–4

    Google Scholar 

  39. Seal A, Ganguly S, Bhattacharjee D, Nasipuri M, Basu DK (2013) Automated thermal face recognition based on minutiae extraction. arXiv preprint arXiv:1309.1000

  40. Lupascu CA, Tegolo D, Trucco E (2010) Fabc: retinal vessel segmentation using adaboost. IEEE Trans Inf Technol Biomed 14(5):1267–1274

    Article  Google Scholar 

  41. Saini I, Singh D, Khosla A (2013) QRS detection using k-nearest neighbor algorithm (KNN) and evaluation on standard ecg databases. J Adv Res 4(4):331–344

    Article  Google Scholar 

  42. Ross A, Jain A (2003) Information fusion in biometrics. Pattern Recogn Lett 24(13):2115–2125

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. PDPM Indian Institute of Information Technology, Design and Manufacturing, Jabalpur, Dumna Airport Road, P.O.: Khamaria, Jabalpur, Madhya Pradesh, 482005, India

    Karnati Mohan & Ayan Seal

Authors
  1. Karnati Mohan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ayan Seal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ayan Seal .

Editor information

Editors and Affiliations

  1. School of Computer Science, University of Petroleum and Energy Studies, Dehradun, India

    Manish Prateek

  2. Department of Informatics, School of Computer Science, University of Petroleum and Energy Studies, Dehradun, India

    T. P. Singh

  3. Department of Informatics, School of Computer Science, University of Petroleum and Energy Studies, Dehradun, India

    Tanupriya Choudhury

  4. Edge Hill University, Ormskirk, UK

    Hari Mohan Pandey

  5. Duy Tan University, Da Nang, Vietnam

    Nguyen Gia Nhu

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mohan, K., Seal, A. (2021). Deception Detection on “Bag-of-Lies”: Integration of Multi-modal Data Using Machine Learning Algorithms. In: Prateek, M., Singh, T.P., Choudhury, T., Pandey, H.M., Gia Nhu, N. (eds) Proceedings of International Conference on Machine Intelligence and Data Science Applications. Algorithms for Intelligent Systems. Springer, Singapore. https://doi.org/10.1007/978-981-33-4087-9_38

Download citation

Publish with us

Policies and ethics

Search

Navigation