Multimedia story creation on social networks
Introduction
Nowadays, Online Social Networks (OSNs) provide users an interactive platform to generate content of heterogeneous nature, especially multimedia data, for many different purposes (e.g. for commenting events and facts, expressing personal opinions on specific themes, sharing moments of everyday life and so on), allowing in this way millions of people to create online profiles and share personal information within vast communities of individuals.
Modern OSNs can be decline theirselves either in systems like Twitter and Facebook – explicitly designed for social interactions – or in terms of applications such as YouTube, Flickr, Instagram and Last.fm targeted at more specific aims such as the sharing of multimedia data and, at the same time, the exhibition of the extensive level of indirect social interaction with their shared content. There are also other kinds of social networks designed to share comments and opinions on specific arguments (e.g. Yelp, IMDB, etc.), to suggest and rate places of interest (e.g. TripAdvisor, Foursquare, etc.) or to create social environments for supporting particular tasks (e.g., the search of a job as in LinkedIn, the answer to research questions as in ResearchGate, etc.), just to mention the most popular ones.
It is noteworthy the fact that OSNs, both for amount of data they produce and the high peace of streaming represent, without any doubt, the essence of Big Data. Moreover, this new availability of social media data allows new opportunities to investigate and analyze social dynamics within these environments. In such a context, Social Network Analysis (SNA) has been used to understand what are the properties of OSNs in supporting a wide range of applications: information retrieval, recommendation, summarization, viral marketing, event recognition, expert finding, community detection, user profiling, security and social data privacy, etc. [[1], [2]].
In the last years, Visual Analytics – an emerging branch of SNA – has received significant attention: different visual analytics methods have been proposed to explore wide-range structured and unstructured data coming from social media. More in details, visual analytics focuses on “analytical reasoning facilitated by interactive visual interfaces” [3] and it is a multidisciplinary field that combines several issues as visualization, human factors, and data analysis [4].
One of the most fascinating possibility of visual analytics is the application of summarization techniques in order to simplify the user data gathering task of social media content, as an example of collecting useful information regarding a given topic. The summarization process from OSNs can be considered a “distilling” process of the most important information from a variety of logically related sources, in order to provide a brief and significant version (usually respect to a given argument) of the social media content.
The heterogeneity of the user generated content leads to the following result of the summarization process: a multimedia story, i.e. a sort of summary integrating different kinds of multimedia data (e.g. images, videos, audios, texts, etc.).
Let us consider, for instance, the typical behavior of a user that aims at searching specific social media content (e.g., photos posted on Flickr or video on Youtube) related to a specific event (e.g., New year’s day in London) described by a set of keywords (e.g. London, new year’s day) and concerning a given topic (e.g., holidays). Once determined the most important objects composing the summary, they have to be properly organized in a multimedia story according to some user preferences and needs and delivered to final users.
In this work, we propose an original summarization technique of social media content applied to OSNs for multimedia story creation. In particular, for each Multimedia Social Network (MuSN) – i.e. an OSN focusing on the management and sharing of multimedia information – we use an graph-based modeling approach and exploit influence analysis methodologies to detect the most important multimedia objects related to one or more topics of interest. Consecutively, from the list of candidate objects we obtain a multimedia summary from a summarization model and an heuristics that consider several properties such as Priority (w.r.t. user keywords), Continuity, Variety and not Repetitiveness. The summary objects are finally arranged in a multimedia story and presented/delivered to final users.
The paper is organized as follows: Section 2 describes the state of the art of the summarization of social media content issue. Section 3 details the proposed summarization process. Sections 4 Multimedia social network modeling and building, 5 Story creation explain the MuSN model and the Summarization model with the proposed influence analysis strategy and used heuristics, respectively. Section 6 depicts the architecture of our summarize, while Section 7 presents the experimental results. Finally, conclusions and future work are provided in Section 7.
Section snippets
Related work
Nowadays large amounts of data are continuously produced by several heterogeneous sources such as OSNs, news feeds and so on. The browsing of these large quantities of data has increased attention on the “summarization” problem, introduced by Luhn [5], that has the aim to produce and present a short and condensed version of a set of documents.
As it is shown in [6], the summarization process can be addressed by the following three steps: firstly several information sources have been analyzed for
The process for social media content summarization at a glance
The summarization process is composed of the following steps:
- 1.
Data Crawling and Scraping: In this step, information related to user generated multimedia content and performed actions within a social environment is periodically captured by means of OSN APIs and logs. The data extracted through this process are then stored in a proper area, namely Staging Area.
- 2.
MuSN Building: Data from staging area are used for building and updating our MuSNs (i.e., one for each considered social network). It can
Multimedia social network modeling and building
As already described, our model permits to define in an effective way any kind of relationships in any type of OSNs. For the purposes of this work, we focus our attention on MuSNs, i.e. particular online social networks in which the user generated content mainly consists of multimedia data such as images, videos, texts and audios.
Popular example of MuSNs are: YouTube, Flickr, Instagram, Last.fm. However, a MuSN is always characterized at least by two different entities: Users — i.e. persons and
Story creation
Generally speaking, we want to determine the most important objects of a MuSN for summarization purposes. In a first step, we face this problem exploiting an Influence Maximization (IM) algorithm that allows to obtain a set of suitable candidates of a summary (or “influentials”), together with the related overall social importance w.r.t. a topic. We successively apply a summarization algorithm on the influentials in order to generate a summary following a set of optimization criteria.
For the
System architecture and implementation
Fig. 1 reports an overview of the proposed summarizer system. We distinguish the following main components that provide an effective support to the different steps of the summarization process as detailed in Section 3. In particular, the main components are: the Data Crawler — that collects information about users, the related generated content and interactions among users and between users and content and then store such information into a Staging Area: the MuSN Builder — that builds the
Experiments and results
As dataset for our experiments, we have used the Yahoo Flickr Creative Commons 100 Million data (YFCC100M)1multimedia collection, provided by Yahoo in 2014. We have thus instantiated the MuSN related to YFCC100M using Flickr basic relationships (i.e., publishing, following, visualizations, comments, favorites, etc.) together with the related multimedia data (images). All information integrating those presented in the dataset were retrieved using Flickr APIs.2
Conclusions and future work
This study aimed at proposing an original multimedia summarization technique for OSNs. In particular a novel model for MuSNs was introduced – i.e. OSNs focusing on the management and sharing of multimedia information – exploiting an graph based approach to represent the different kinds of entities and relationships of the various OSNs. In addition, we adopted influence analysis methodologies to determine the most important multimedia objects specific to one or more topics of interest. We thus
Flora Amato received Ph.D. degree in computer science and engineering from the University of Naples “Federico II”, Italy. She, is currently Assistant Professor at the Department of Electrical Engineering and Information Technology of University of Napoli Federico II where she carries out her research activity since 2006. Her research activities involve both theoretical and experimental researches, and mainly concern Formal Modeling , Verification Techniques, Knowledge Management, Information
References (34)
- et al.
Multimedia news summarization in search
ACM Trans. Intell. Syst. Technol. (TIST)
(2016) - et al.
The priority curve algorithm for video summarization
Inf. Syst.
(2006) - et al.
Leveraging multimodal information for event summarization and concept-level sentiment analysis
Knowl.-Based Syst.
(2016) - et al.
S-InTime: A social cloud analytical service oriented system
Future Gener. Comput. Syst.
(2018) - et al.
Diffusion centrality: A paradigm to maximize spread in social networks
Artificial Intelligence
(2016) Social Network Analysis
(2012)- et al.
Evolutionary network analysis: A survey
ACM Comput. Surv. (CSUR)
(2014) Illuminating the Path:[The Research and Development Agenda for Visual Analytics]
(2005)- et al.
A survey on visual analytics of social media data
IEEE Trans. Multimed.
(2016) The automatic creation of literature abstracts
IBM J. Res. Dev.
(1958)
The challenges of automatic summarization
Computer
A survey of text summarization techniques
iwin: A summarizer system based on a semantic analysis of web documents
A novel approach for automatic text analysis and generation for the cultural heritage domain
Multimedia Tools Appl.
Summarizing multimedia content
Beyond audio and video retrieval: towards multimedia summarization
A multimedia summarizer integrating text and images
Cited by (34)
Extracting video games rating labels from transcript files
2021, Internet of Things (Netherlands)Citation Excerpt :The most used algorithm to extract similarity from vectors are Cosine Similarity, Pearson Correlation, and Euclidean Distance [13]. Maisto et al. [14] convert similarity measures in a matrix of text similarities in order to use network classification algorithms (Modularity) to classify texts or extract a set of objects from the network [15]. Feature selection algorithms allow choosing words that best represent the document in its vector form.
Neoaltar: An interactive multimedia day of the dead experience
2020, HeliyonCitation Excerpt :Amato et al. 2018a,b), this leads to the creation of a multimedia story in accordance with predefined constraints. The summarization process and algorithm presented by Amato et al. 2018a,b poses an opportunity to create personalized altars based on information that gets automatically gather, following certain constraints, either from social media profiles or real-time user-generated data. Other works include mobile applications, games, interactive books, and educational applications, where the user can tap on objects in the altar and description and the meaning of each item is displayed; in these cases, technology takes over and the essence of the traditional altar is not present, since all physical elements are eliminated and replaced by graphics on a screen.
Internet of Knowledge
2020, Future Generation Computer SystemsStudy of hospitalization for retinal surgery using classification algorithms
2023, ACM International Conference Proceeding SeriesMachine learning algorithms to study the hospitalization after cesarean section: a multicenter analysis
2023, ACM International Conference Proceeding SeriesProtoLife: Enhancing Mobile Multimedia Narratives Through Prototyping Techniques
2023, Communications in Computer and Information Science
Flora Amato received Ph.D. degree in computer science and engineering from the University of Naples “Federico II”, Italy. She, is currently Assistant Professor at the Department of Electrical Engineering and Information Technology of University of Napoli Federico II where she carries out her research activity since 2006. Her research activities involve both theoretical and experimental researches, and mainly concern Formal Modeling , Verification Techniques, Knowledge Management, Information Extraction and Integration. She is author of more of 70 research papers, published on International Journals and Conference Proceedings.
Aniello Castiglione received the Ph.D. degree in Computer Science from the University of Salerno. Actually he is an adjunct professor at the University of Salerno (Italy) and at the University of Naples “Federico II”. His current research interests include Information Forensics, Digital Forensics, Security and Privacy on Cloud, Communication Networks, and Applied Cryptography. He published more than 140 papers in international journals and conferences. He served as Program Chair and TPC Member in around 90 international conferences. He acted as a Guest Editor in several journals and serves as Editor in several editorial boards of international journals.
Fabio Mercorio is Assistant Professor of Computer Science at the University of Milan-Bicocca. He holds a PhD in Computer Science and Application in 2012 at the University of L’Aquila, Italy. His research interests include Artificial Intelligence Planning and Knowledge Discovery. His main contribution is in the design and development of an automated algorithm for performing Data Quality Analysis and Cleansing tasks through model-checking. Since 2011 he has been research fellow at CRISP, where he has been involved in many national and International research projects related to Data Management and Knowledge Discovery for supporting the decision making activities.
Mario Mezzanzanica is Associate Professor of “Information Systems” at the University of Milan Bicocca. He is the Scientific Director of the “CRISP” centre — Inter University Research Centre on public services. His research interests include Information Systems, Databases, Artificial Intelligence, Business Intelligence and Knowledge Discovery. He is member of the international journals editorial board in the field of Artificial Intelligence and Information Systems. He has also been involved in several technical and scientific committees activated by Public Institutions, aimed at studying new models and methodologies to design, monitor and evaluation of innovation projects, with relevant impacts on ICT-based public services.
Vincenzo Moscato received the Laurea degree cum laude and a Ph.D. degree in computer science and engineering both from the University of Naples “Federico II”. He is currently an Associate Professor of Data Base and Computer Engineering at Department of Electrical Engineering and Information Technologies of University of Naples “Federico II”. His current research interests lie in the area of multimedia, knowledge representation and management and Big Data. He was involved in several international, national and local research projects and at present is an author of more than eighty publications on international journal and conference proceedings.
Antonio Picariello received the Laurea degree in electronics engineering and a Ph.D. degree in computer science and engineering both from the University of Naples “Federico II”. In 1999, he joined the Department of Computer Science and Systems, University of Naples “Federico II” and is currently an Associate Professor of multimedia databases and computer engineering and the Director of CINI-ITEM National Laboratory for Multimedia and Network Information Technologies. His current research interests include knowledge extraction and management, multimedia integration, image and video databases and multimedia social network analysis.
Giancarlo Sperli received the Master degree in computer science and engineering from the University of Naples “Federico II”, Italy. Now, he is PH.D. Student at the University of Naples “Federico II”. His research activities are focused on the following topics: Big Data, Multimedia Social Network and Cyber Security, Multimedia and knowledge representation and management.