Tankut Akgul
Ali C. Begen
ABSTRACT
The growing prevalence of visual disinformation has become an important problem to solve nowadays. Cheapfake is a new term used for the altered media generated by non-AI techniques. In their recent COSMOS work, the authors developed a self-supervised training strategy that detected whether different captions for a given image were out-of-context, meaning that even though pointing to the same object(s) in the image, the captions implied different meanings. In this paper, we propose four methods to improve the detection accuracy of COSMOS. These methods range from differential sensing and fake-or-fact checking that detect contradicting or fake captions to object-caption matching and threshold adjustment that modify the baseline algorithm for improved accuracy.
Supplemental Material
Available for Download
Supplemental material.
- S. Aneja, C. Bregler, and M. Nießner. COSMOS: Catching out-of-context misinformation with self-supervised learning. https://arxiv.org/abs/2101.06278, 2021.Google Scholar
- S. Aneja, C. Midoglu, D.-T. Dang-Nguyen, M. A. Riegler, P. Halvorsen, M. Niessner, B. Adsumilli, and C. Bregler. MMSys'21 grand challenge on detecting cheapfakes. https://arxiv.org/abs/2107.05297, 2021.Google Scholar
- D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. St. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, B. Strope, and R. Kurzweil. Universal sentence encoder for English. In Proc. Conf. Empirical Methods in Natural Language Processing: System Demonstrations, pages 169--174, Brussels, Belgium, 2018. Association for Computational Linguistics.Google ScholarCross Ref
- P. Fialho, L. Coheur, and P. Quaresma. To BERT or not to BERT dealing with possible bert failures in an entailment task. In Information Processing and Management of Uncertainty in Knowledge-Based Systems, pages 734--747. Springer International Publishing, 2020.Google ScholarCross Ref
- K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask R-CNN. In IEEE Int. Conf. Computer Vision (ICCV), 2017.Google Scholar
- J. Scott Brennen, F. M. Simon, P. N. Howard, and R. K. Nielsen. Types, sources, and claims of COVID-19 misinformation. [Online] Available: http://www.primaonline.it/wp-content/uploads/2020/04/COVID-19_reuters.pdf. Accessed on July 19, 2021.Google Scholar
- Y. Mirsky and W. Lee. The creation and detection of deepfakes: A survey. ACM Comput. Surv., 54(1), Jan. 2021.Google Scholar
- B. Paris and J. Donovan. Deepfakes and Cheap fakes: The Manipulation of Audio and Visual Evidence. [Online] Available: https://datasociety.net/library/deepfakes-and-cheap-fakes/. Accessed on July 19, 2021.Google Scholar
- N. Reimers and I. Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proc. of the 2019 Conf. Empirical Methods in Natural Language Processing and the 9th Int. Joint Conf. Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, Nov. 2019.Google ScholarCross Ref
- N. Schick. Don’t underestimate the cheapfake. [Online] Available: https://www.technologyreview.com/2020/12/22/1015442/cheapfakes-more-political-damage-2020-election-than-deepfakes/. Accessed on July 19, 2021.Google Scholar
- B. Wang and C.-C. J. Kuo. SBERT-WK: a sentence embedding method by dissecting BERT-based word models. IEEE/ACM Trans. Audio, Speech, and Language Processing, 28:2146--2157, 2020.Google ScholarDigital Library
Index Terms
- COSMOS on Steroids: a Cheap Detector for Cheapfakes
Recommendations
A Combination of Visual-Semantic Reasoning and Text Entailment-based Boosting Algorithm for Cheapfake Detection
MM '22: Proceedings of the 30th ACM International Conference on MultimediaMisuse of real photographs with conflicting image captions in news items is one case of out-of-context (OOC) misuse of media. To detect out-of-context given pair of news (i.e., captions) and attached image, people should determine the truthfulness of ...
A Multi-Stage Deep Learning Approach Incorporating Text-Image and Image-Image Comparisons for Cheapfake Detection
ICMR '24: Proceedings of the 2024 International Conference on Multimedia RetrievalThe advancement of multimedia and artificial intelligence (AI) technologies has dismantled the barriers of information sharing, yet it has also ushered in a double-edged sword: a surge in the spread of fake information. In this context, there is a ...
SCAN: A shared causal attention network for adverse drug reactions detection in tweets
AbstractTwitter is a popular social media site on which people post millions of Tweets every day. As patients often share their experiences with drugs on Twitter, Tweets can also be considered as a rich alternative source of adverse drug ...
Comments