Elsevier

Future Generation Computer Systems

Volume 86, September 2018, Pages 412-420
Future Generation Computer Systems

Multimedia story creation on social networks

https://doi.org/10.1016/j.future.2018.04.006Get rights and content

Highlights

  • We describe a novel multimedia summarization technique from Online Social Networks.

  • We model each Multimedia Social Network (MSN) using an hypergraph based approach.

  • We determine the most important multimedia objects by using a bio-inspired strategy.

  • We generate summaries with priority, continuity, variety and not receptiveness features.

  • We produce summaries combining a novel summarization model with a heuristic strategy.

Abstract

The paper aims at proposing an original summarization technique from Online Social Networks (OSNs) for multimedia stories’ creation. In particular, for each Multimedia Social Network (MuSN) – i.e. an OSN focusing on the management and sharing of multimedia information – we leverage a graph-based modeling approach and exploit influence analysis methodologies to detect the most important multimedia objects related to one or more topics of interest. Consecutively, from the list of candidate objects we obtain a multimedia summary on the basis of a novel summarization model and some heuristics, whose purpose is to generate multimedia stories with priority (w.r.t. some user keywords), continuity, variety and not receptiveness features. The effectiveness of the proposed approach is shown by the performed experiments on Flickr.

Introduction

Nowadays, Online Social Networks (OSNs) provide users an interactive platform to generate content of heterogeneous nature, especially multimedia data, for many different purposes (e.g. for commenting events and facts, expressing personal opinions on specific themes, sharing moments of everyday life and so on), allowing in this way millions of people to create online profiles and share personal information within vast communities of individuals.

Modern OSNs can be decline theirselves either in systems like Twitter and Facebook – explicitly designed for social interactions – or in terms of applications such as YouTube, Flickr, Instagram and Last.fm targeted at more specific aims such as the sharing of multimedia data and, at the same time, the exhibition of the extensive level of indirect social interaction with their shared content. There are also other kinds of social networks designed to share comments and opinions on specific arguments (e.g. Yelp, IMDB, etc.), to suggest and rate places of interest (e.g. TripAdvisor, Foursquare, etc.) or to create social environments for supporting particular tasks (e.g., the search of a job as in LinkedIn, the answer to research questions as in ResearchGate, etc.), just to mention the most popular ones.

It is noteworthy the fact that OSNs, both for amount of data they produce and the high peace of streaming represent, without any doubt, the essence of Big Data. Moreover, this new availability of social media data allows new opportunities to investigate and analyze social dynamics within these environments. In such a context, Social Network Analysis (SNA) has been used to understand what are the properties of OSNs in supporting a wide range of applications: information retrieval, recommendation, summarization, viral marketing, event recognition, expert finding, community detection, user profiling, security and social data privacy, etc. [[1], [2]].

In the last years, Visual Analytics – an emerging branch of SNA – has received significant attention: different visual analytics methods have been proposed to explore wide-range structured and unstructured data coming from social media. More in details, visual analytics focuses on “analytical reasoning facilitated by interactive visual interfaces” [3] and it is a multidisciplinary field that combines several issues as visualization, human factors, and data analysis [4].

One of the most fascinating possibility of visual analytics is the application of summarization techniques in order to simplify the user data gathering task of social media content, as an example of collecting useful information regarding a given topic. The summarization process from OSNs can be considered a “distilling” process of the most important information from a variety of logically related sources, in order to provide a brief and significant version (usually respect to a given argument) of the social media content.

The heterogeneity of the user generated content leads to the following result of the summarization process: a multimedia story, i.e. a sort of summary integrating different kinds of multimedia data (e.g. images, videos, audios, texts, etc.).

Let us consider, for instance, the typical behavior of a user that aims at searching specific social media content (e.g., photos posted on Flickr or video on Youtube) related to a specific event (e.g., New year’s day in London) described by a set of keywords (e.g. London, new year’s day) and concerning a given topic (e.g., holidays). Once determined the most important objects composing the summary, they have to be properly organized in a multimedia story according to some user preferences and needs and delivered to final users.

In this work, we propose an original summarization technique of social media content applied to OSNs for multimedia story creation. In particular, for each Multimedia Social Network (MuSN) – i.e. an OSN focusing on the management and sharing of multimedia information – we use an graph-based modeling approach and exploit influence analysis methodologies to detect the most important multimedia objects related to one or more topics of interest. Consecutively, from the list of candidate objects we obtain a multimedia summary from a summarization model and an heuristics that consider several properties such as Priority (w.r.t. user keywords), Continuity, Variety and not Repetitiveness. The summary objects are finally arranged in a multimedia story and presented/delivered to final users.

The paper is organized as follows: Section 2 describes the state of the art of the summarization of social media content issue. Section 3 details the proposed summarization process. Sections 4 Multimedia social network modeling and building, 5 Story creation explain the MuSN model and the Summarization model with the proposed influence analysis strategy and used heuristics, respectively. Section 6 depicts the architecture of our summarize, while Section 7 presents the experimental results. Finally, conclusions and future work are provided in Section 7.

Section snippets

Related work

Nowadays large amounts of data are continuously produced by several heterogeneous sources such as OSNs, news feeds and so on. The browsing of these large quantities of data has increased attention on the “summarization” problem, introduced by Luhn [5], that has the aim to produce and present a short and condensed version of a set of documents.

As it is shown in [6], the summarization process can be addressed by the following three steps: firstly several information sources have been analyzed for

The process for social media content summarization at a glance

The summarization process is composed of the following steps:

  • 1.

    Data Crawling and Scraping: In this step, information related to user generated multimedia content and performed actions within a social environment is periodically captured by means of OSN APIs and logs. The data extracted through this process are then stored in a proper area, namely Staging Area.

  • 2.

    MuSN Building: Data from staging area are used for building and updating our MuSNs (i.e., one for each considered social network). It can

Multimedia social network modeling and building

As already described, our model permits to define in an effective way any kind of relationships in any type of OSNs. For the purposes of this work, we focus our attention on MuSNs, i.e. particular online social networks in which the user generated content mainly consists of multimedia data such as images, videos, texts and audios.

Popular example of MuSNs are: YouTube, Flickr, Instagram, Last.fm. However, a MuSN is always characterized at least by two different entities: Users — i.e. persons and

Story creation

Generally speaking, we want to determine the most important objects of a MuSN for summarization purposes. In a first step, we face this problem exploiting an Influence Maximization (IM) algorithm that allows to obtain a set of suitable candidates of a summary (or “influentials”), together with the related overall social importance w.r.t. a topic. We successively apply a summarization algorithm on the influentials in order to generate a summary following a set of optimization criteria.

For the

System architecture and implementation

Fig. 1 reports an overview of the proposed summarizer system. We distinguish the following main components that provide an effective support to the different steps of the summarization process as detailed in Section 3. In particular, the main components are: the Data Crawler  — that collects information about users, the related generated content and interactions among users and between users and content and then store such information into a Staging Area: the MuSN Builder  — that builds the

Experiments and results

As dataset for our experiments, we have used the Yahoo Flickr Creative Commons 100 Million data (YFCC100M)1multimedia collection, provided by Yahoo in 2014. We have thus instantiated the MuSN related to YFCC100M using Flickr basic relationships (i.e., publishing, following, visualizations, comments, favorites, etc.) together with the related multimedia data (images). All information integrating those presented in the dataset were retrieved using Flickr APIs.2

Conclusions and future work

This study aimed at proposing an original multimedia summarization technique for OSNs. In particular a novel model for MuSNs was introduced – i.e. OSNs focusing on the management and sharing of multimedia information – exploiting an graph based approach to represent the different kinds of entities and relationships of the various OSNs. In addition, we adopted influence analysis methodologies to determine the most important multimedia objects specific to one or more topics of interest. We thus

Flora Amato received Ph.D. degree in computer science and engineering from the University of Naples “Federico II”, Italy. She, is currently Assistant Professor at the Department of Electrical Engineering and Information Technology of University of Napoli Federico II where she carries out her research activity since 2006. Her research activities involve both theoretical and experimental researches, and mainly concern Formal Modeling , Verification Techniques, Knowledge Management, Information

References (34)

  • HahnU. et al.

    The challenges of automatic summarization

    Computer

    (2000)
  • NenkovaA. et al.

    A survey of text summarization techniques

  • MoscatoV. et al.

    iwin: A summarizer system based on a semantic analysis of web documents

  • PiccialliF. et al.

    A novel approach for automatic text analysis and generation for the cultural heritage domain

    Multimedia Tools Appl.

    (2017)
  • ModaniN. et al.

    Summarizing multimedia content

  • DingD. et al.

    Beyond audio and video retrieval: towards multimedia summarization

  • d’AciernoA. et al.

    A multimedia summarizer integrating text and images

  • Cited by (34)

    • Extracting video games rating labels from transcript files

      2021, Internet of Things (Netherlands)
      Citation Excerpt :

      The most used algorithm to extract similarity from vectors are Cosine Similarity, Pearson Correlation, and Euclidean Distance [13]. Maisto et al. [14] convert similarity measures in a matrix of text similarities in order to use network classification algorithms (Modularity) to classify texts or extract a set of objects from the network [15]. Feature selection algorithms allow choosing words that best represent the document in its vector form.

    • Neoaltar: An interactive multimedia day of the dead experience

      2020, Heliyon
      Citation Excerpt :

      Amato et al. 2018a,b), this leads to the creation of a multimedia story in accordance with predefined constraints. The summarization process and algorithm presented by Amato et al. 2018a,b poses an opportunity to create personalized altars based on information that gets automatically gather, following certain constraints, either from social media profiles or real-time user-generated data. Other works include mobile applications, games, interactive books, and educational applications, where the user can tap on objects in the altar and description and the meaning of each item is displayed; in these cases, technology takes over and the essence of the traditional altar is not present, since all physical elements are eliminated and replaced by graphics on a screen.

    • Internet of Knowledge

      2020, Future Generation Computer Systems
    • Study of hospitalization for retinal surgery using classification algorithms

      2023, ACM International Conference Proceeding Series
    • ProtoLife: Enhancing Mobile Multimedia Narratives Through Prototyping Techniques

      2023, Communications in Computer and Information Science
    View all citing articles on Scopus

    Flora Amato received Ph.D. degree in computer science and engineering from the University of Naples “Federico II”, Italy. She, is currently Assistant Professor at the Department of Electrical Engineering and Information Technology of University of Napoli Federico II where she carries out her research activity since 2006. Her research activities involve both theoretical and experimental researches, and mainly concern Formal Modeling , Verification Techniques, Knowledge Management, Information Extraction and Integration. She is author of more of 70 research papers, published on International Journals and Conference Proceedings.

    Aniello Castiglione received the Ph.D. degree in Computer Science from the University of Salerno. Actually he is an adjunct professor at the University of Salerno (Italy) and at the University of Naples “Federico II”. His current research interests include Information Forensics, Digital Forensics, Security and Privacy on Cloud, Communication Networks, and Applied Cryptography. He published more than 140 papers in international journals and conferences. He served as Program Chair and TPC Member in around 90 international conferences. He acted as a Guest Editor in several journals and serves as Editor in several editorial boards of international journals.

    Fabio Mercorio is Assistant Professor of Computer Science at the University of Milan-Bicocca. He holds a PhD in Computer Science and Application in 2012 at the University of L’Aquila, Italy. His research interests include Artificial Intelligence Planning and Knowledge Discovery. His main contribution is in the design and development of an automated algorithm for performing Data Quality Analysis and Cleansing tasks through model-checking. Since 2011 he has been research fellow at CRISP, where he has been involved in many national and International research projects related to Data Management and Knowledge Discovery for supporting the decision making activities.

    Mario Mezzanzanica is Associate Professor of “Information Systems” at the University of Milan Bicocca. He is the Scientific Director of the “CRISP” centre — Inter University Research Centre on public services. His research interests include Information Systems, Databases, Artificial Intelligence, Business Intelligence and Knowledge Discovery. He is member of the international journals editorial board in the field of Artificial Intelligence and Information Systems. He has also been involved in several technical and scientific committees activated by Public Institutions, aimed at studying new models and methodologies to design, monitor and evaluation of innovation projects, with relevant impacts on ICT-based public services.

    Vincenzo Moscato received the Laurea degree cum laude and a Ph.D. degree in computer science and engineering both from the University of Naples “Federico II”. He is currently an Associate Professor of Data Base and Computer Engineering at Department of Electrical Engineering and Information Technologies of University of Naples “Federico II”. His current research interests lie in the area of multimedia, knowledge representation and management and Big Data. He was involved in several international, national and local research projects and at present is an author of more than eighty publications on international journal and conference proceedings.

    Antonio Picariello received the Laurea degree in electronics engineering and a Ph.D. degree in computer science and engineering both from the University of Naples “Federico II”. In 1999, he joined the Department of Computer Science and Systems, University of Naples “Federico II” and is currently an Associate Professor of multimedia databases and computer engineering and the Director of CINI-ITEM National Laboratory for Multimedia and Network Information Technologies. His current research interests include knowledge extraction and management, multimedia integration, image and video databases and multimedia social network analysis.

    Giancarlo Sperli received the Master degree in computer science and engineering from the University of Naples “Federico II”, Italy. Now, he is PH.D. Student at the University of Naples “Federico II”. His research activities are focused on the following topics: Big Data, Multimedia Social Network and Cyber Security, Multimedia and knowledge representation and management.

    View full text