Abstract
In this paper we present a novel interactive multimodal learning system, which facilitates search and exploration in large networks of social multimedia users. It allows the analyst to identify and select users of interest, and to find similar users in an interactive learning setting. Our approach is based on novel multimodal representations of users, words and concepts, which we simultaneously learn by deploying a general-purpose neural embedding model. The usefulness of the approach is evaluated using artificial actors, which simulate user behavior in a relevance feedback scenario. Multiple experiments were conducted in order to evaluate the quality of our multimodal representations and compare different embedding strategies. We demonstrate the capabilities of the proposed approach on a multimedia collection originating from the violent online extremism forum Stormfront, which is particularly interesting due to the high semantic level of the discussions it features.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Barthel, K.U., Hezel, N., Mackowiak, R.: Navigating a graph of scenes for exploring large video collections. In: Tian, Q., Sebe, N., Qi, G.-J., Huet, B., Hong, R., Liu, X. (eds.) MMM 2016. LNCS, vol. 9517, pp. 418–423. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-27674-8_43
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. TACL 5, 135–146 (2017)
Chen, J.J., Ngo, C.W., Feng, F.L., Chua, T.S.: Deep understanding of cooking procedure for cross-modal recipe retrieval. In: ACM MM 2018, pp. 1020–1028 (2018)
Conway, M.: Determining the role of the internet in violent extremism and terrorism. In: Violent Extremism Online: New Perspectives on Terrorism and the Internet, p. 123 (2016)
van der Corput, P., van Wijk, J.J.: ICLIC: interactive categorization of large image collections. In: IEEE PacificVis 2016, pp. 152–159 (2016)
van der Corput, P., van Wijk, J.J.: Comparing personal image collections with picturevis. In: Computer Graphics Forum, vol. 36, no. 3, pp. 295–304 (2017)
Grover, A., Leskovec, J.: Node2vec: scalable feature learning for networks. In: ACM KDD 2016, pp. 855–864 (2016)
Lokoč, J., Kovalčík, G., Souček, T.: Revisiting SIRET video retrieval tool. In: Schoeffmann, K., et al. (eds.) MMM 2018. LNCS, vol. 10705, pp. 419–424. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73600-6_44
Lokoč, J., Bailer, W., Schoeffmann, K., Muenzer, B., Awad, G.: On influential trends in interactive video retrieval: video browser showdown 2015–2017. IEEE TMM 20(12), 3361–3376 (2018)
Martin, N., Maes, H.: Multivariate Analysis. Academic Press, Cambridge (1979)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: NIPS 2013, pp. 3111–3119 (2013)
Odijk, D., Meij, E., de Rijke, M.: Feeding the second screen: semantic linking based on subtitles. In: OAIR 2013, pp. 9–16 (2013)
Pennington, J., Socher, R., Manning, C.: Glove: global vectors for word representation. In: EMNLP 2014, pp. 1532–1543 (2014)
Qi, M., Wang, Y., Li, A.: Online cross-modal scene retrieval by binary representation and semantic graph. In: ACM MM 2017, pp. 744–752 (2017)
Rossetto, L., Amiri Parian, M., Gasser, R., Giangreco, I., Heller, S., Schuldt, H.: Deep learning-based concept detection in vitrivr. In: Kompatsiaris, I., Huet, B., Mezaris, V., Gurrin, C., Cheng, W.-H., Vrochidis, S. (eds.) MMM 2019. LNCS, vol. 11296, pp. 616–621. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05716-9_55
Rudinac, S., Gornishka, I., Worring, M.: Multimodal classification of violent online political extremism content with graph convolutional networks. In: Thematic Workshops of ACM MM 2017, pp. 245–252 (2017)
Salton, G., Buckley, C.: Term-weighting approaches in automatic text retrieval. Inf. Process. Manag. 24(5), 513–523 (1988)
Snoek, C.G.M., et al.: Mediamill at TRECVID 2013: searching concepts, objects, instances and events in video. In: TRECVID Workshop (2013)
Wang, B., Yang, Y., Xu, X., Hanjalic, A., Shen, H.T.: Adversarial cross-modal retrieval. In: ACM MM 2017, pp. 154–162 (2017)
Worring, M., Koelma, D., Zahálka, J.: Multimedia pivot tables for multimedia analytics on image collections. In: IEEE TMM 2016, vol. 18, no. 11, pp. 2217–2227, November 2016
Wu, L.Y., Fisch, A., Chopra, S., Adams, K., Bordes, A., Weston, J.: Starspace: embed all the things! In: AAAI 2018, pp. 5569–5577 (2018)
Wu, Y., Wang, S., Huang, Q.: Learning semantic structure-preserved embeddings for cross-modal retrieval. In: ACM MM 2018, pp. 825–833 (2018)
Yang, Y., Luo, Y., Chen, W., Shen, F., Shao, J., Shen, H.T.: Zero-shot hashing via transferring supervised knowledge. In: ACM MM 2016, pp. 1286–1295 (2016)
Zahálka, J., Rudinac, S., Jónsson, B.T., Koelma, D.C., Worring, M.: Blackthorn: large-scale interactive multimodal learning. IEEE TMM 20(3), 687–698 (2018)
Zahálka, J., Rudinac, S., Worring, M.: Analytic quality: evaluation of performance and insight in multimedia collection analysis. In: ACM MM 2015, pp. 231–240 (2015)
Zahálka, J., Rudinac, S., Worring, M.: Interactive multimodal learning for venue recommendation. IEEE TMM 17(12), 2235–2244 (2015)
Acknowledgments
This project has received funding from the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no. 312827 (NoE VOX-Pol).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Gornishka, I., Rudinac, S., Worring, M. (2020). Interactive Search and Exploration in Discussion Forums Using Multimodal Embeddings. In: Ro, Y., et al. MultiMedia Modeling. MMM 2020. Lecture Notes in Computer Science(), vol 11962. Springer, Cham. https://doi.org/10.1007/978-3-030-37734-2_32
Download citation
DOI: https://doi.org/10.1007/978-3-030-37734-2_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37733-5
Online ISBN: 978-3-030-37734-2
eBook Packages: Computer ScienceComputer Science (R0)