An architecture for non-linear discovery of aggregated multimedia document web search results

Introduction

Traditionally, the web contains only the textual content (Bianchi-Berthouze et al., 2003). The progressive easy access to the internet has transformed the web into an infinitely complex virtual organism consisting of immense multimedia content (Batrinca & Treleaven, 2015). The format of the information is now extremely varied. The individual bits of data coming from blogs, articles, web services, picture galleries, etc., are resulting in exponential growth of multimedia data on the web (Batrinca & Treleaven, 2015; Rashid & Bhatti, 2017). The web is becoming the most ubiquitous platform ever since its birth and has increased in both quantity and quality (Taheri et al., 2018). In 2009, less than 1 petabyte of digital data was created daily (Kumar & Ogunmola, 2020). It grew to approximately 2.5 exabytes in 2012 and reached 4.4 zettabytes in 2013. On the web, the digital data in different formats created, replicated, and consumed exponentially (Oussous et al., 2018). It is doubling every 2 years. By 2015, digital data grew to 8 zettabytes, and the volume of data will reach 40 zettabytes by the end of 2020 (Oussous et al., 2018).

Keywords-based general web search engines have made early efforts to provide access to multimedia information (Lewandowski, 2008). These search engines required a user to enter one or a few keywords, and the search engines produced the relevant results in a short time (Lewandowski, 2008). Kerne & Smith (2004) first discussed a new search paradigm called information discovery. They elaborated discovery as a long journey of search that begins with a vague description of a problem, may have an articulated set of criteria during which a searcher specify a query and evaluate the returned information surrogates, and may continue iteratively by re-evaluating the result sets and forming a sense of desired results. Marchionini (2006) gives the same idea in a broader perspective by categorizing the search paradigm into an exploratory by incorporating not only lookup searches but learning and investigation activities. Adequate support in the users’ search leads to the discovery of new information items.

In contrast, many current search systems assume an exploratory search process as a series of homogeneous steps of submitting a query and consulting search results. Research in information seeking has shown that users go through discrete phases in their search journey, from exploring and identifying preliminary information to refining and narrowing their information needs and search strategies to finalize the search. It is reported as a highly complex problem bridging the different areas of information seeking, interactive information retrieval, and user interface design (Gäde et al., 2015). Moreover, the increasing amount of heterogeneous content on the web has transformed user needs from simple lookup-based queries to broader exploratory queries, requiring the diverse heterogeneous contents to satisfy the desired information needs (Rashid & Bhatti, 2017).

Several studies indicate that more intricate tasks resulted in a diversity of the information sought and more varied approaches to information seeking (Campbell, 2013). Today, to find interesting multimedia content, an enormous number of users use search engines (Deldjoo et al., 2018). It has changed users’ information need from textual to multi-modal (audio, image, and video) searching. Approximately 40–50% of users engage in dynamic and unplanned nature of web multimedia searches (Tseng, Tjondronegoro & Spink, 2009). When the information need is ambiguous and dynamic (e.g., in exploratory search), people often consult more multimedia search results (Bron et al., 2013). The need for multimedia documents, in this case, increases to 58% (Tseng, Tjondronegoro & Spink, 2009).

As human information needs and search tasks become complex, the users have to collect and assemble information from diverse information sources. The goal is to compose the most appropriate responses to the tasks at hand in the form of multimedia documents (Kopliku, Pinel-Sauvagnat & Boughanem, 2014). A multimedia document is a collection of co-existing heterogeneous multimedia objects sharing the same semantics (Rashid & Bhatti, 2017). Users prefer aggregation of useful multimedia information residing in diverse sources through unified interfaces (Kopliku, Pinel-Sauvagnat & Boughanem, 2014; Rashid & Bhatti, 2017). Similarly, the user interface presenting aggregated contents encouraged participants to view more diversified sources from the search results, and 75% of the participants found this blended approach more comfortable to use (Sushmita, Joho & Lalmas, 2009). The user click-through rate analysis reported approximately 33% on augmented multimedia artifacts and nearly 55% multimedia artifacts were found relevant and useful during information exploration activities (Sushmita et al., 2010). Overall, users explore the multimedia contents 78% of the time to answer their dynamic information needs (Kopliku et al., 2011). Based on recent user behavior in complex information needs, we can easily forecast even more increasing multimedia artifacts consumption from the users in satisfaction of complex information needs and discovering information.

Aggregating disjoint verticals provide access to diverse multimedia documents. A vertical is defined as a specialized assembly of same-typed documents (Bakrola & Gandhi, 2016). This assembly can be media-specific or domain-specific. The former may include media types (e.g., video, blog, image, etc.). The latter may consist of verticals (e.g., travel, shopping, news, etc.). The aggregation process consists of either cross-vertical Aggregated Search (cvAS) or Relational Aggregated Search (RAS). The (cvAS) ignores the relation during retrieval and aggregation of multimedia content. The (RAS) considers the relationships in the multimedia information. Despite the key-role of aggregation in bridging the modality gap, this area of research only received limited attention in the past (Achsas & Nfaoui, 2019). Without substantial creativity, this area of research will soon be abandoned. Our discovery approach aims to bring innovation and creativity in this area of search. We envision bridging the modality gap and shortcomings of current search engines, allowing users to discover multimedia information by aggregating disjoint verticals.

The contributions of our solution are threefold. Firstly, we presented a creative search results aggregation technique using state-of-the-art semantic analysis. Secondly, we enhanced the current search engine shortcomings in information exploration and discovery activities by augmenting non-linear information seeking patterns. Thirdly, we bridged the information modality gap by encoding the search results in various representations. Our proposed solution is the first to address all of the stated challenges of information aggregation, exploration, and discovery.

The rest of the discussion is organized as follows. We discuss the related work in “Related Work”. We highlight the deficiencies in the existing approaches and motivation behind this research in “Problem & Motivation”. We provide the theoretical foundation and formalization of our proposed approach in “Architecture Design: Definition, Formalization & Instantiation”. We present the implementation of the architecture in “Architectural Implementation”. We discuss the experimental results in “Results”. Finally, we compare our approach with state-of-the-art and conclude our discussion in “Comparison & Discussion” and “Conclusion & Future Work”, respectively.

Related work

Problem & motivation

The users’ complex information-seeking behavior is modeled as a non-linear journey requiring adequate support during the navigation of the information space (Ruotsalo et al., 2018). Users’ forage for the information (Bates, 1989). Their complex information needs are not sufficed through the current ideology of returning the most precise information in response to the given queries (Russell-Rose & Tate, 2012). Instead, users pick the most interesting items like barries from various patches of information, providing more information scent ratio to the effort required for examining the information. It results in a non-linear information searching pattern of users (Russell-Rose & Tate, 2012).

The search engines are more tuned towards simple lookup searches favoring precision over recall (Tablan et al., 2015). However, even though they have recognized the users’ multimedia information needs and started to blend some vertical-specific results assembled from the other data sources (Bakrola & Gandhi, 2016). The current practices of presenting information in a linear ranked list of standard results limit information exploration (Rashid & Bhatti, 2017). Furthermore, the integration of the verticals is mostly partial-blended (Rashid & Bhatti, 2017), which may suffice in simple lookup searches when a user knows what to look for; however, this strategy inadequately support complex information exploration and discovery tasks (Tablan et al., 2015). These tasks go beyond simple keyword-based queries. Users often have difficulties in information need expression, and they usually are dynamic (Ruotsalo et al., 2015a). Such tasks require more recall over precision and diversity of information sources (Tablan et al., 2015). It challenges the current practices of displaying the search results belonging to different verticals as disjoint sets (Rashid & Bhatti, 2017).

On the other hand, the search engines remain almost the same as they were about a decade ago. There exist numerous problems (P) with current search engines. The Fig. 1 shows the difference between the Google Search Engine Results Page (SERP) back in 2010 (Sullivan, 2020) and now in 2020 (Google, 2020). The verticals are integrated as disjoint components (P₁). The relationships between multimedia objects are ignored (P₂). The information presented is still displayed as linear lists (P₃). This presentation of the general search engines’ information may suffice for simple lookup tasks but lacks adequacy for complex exploratory and discovery tasks (Klouche et al., 2015). These tasks require increased recall over precision (P₄), information scent (P₅), and sensemaking (P₆). The existing exploration approaches’ deficiencies demand a better mechanism to encode and present the multimedia information for discovery (P₇).

Architecture design: definition, formalization & instantiation

Existing techniques are usually specific to a problem and employed on a particular dataset. Many researchers consider one side of the discovery, such as information diversification, visualization, data-modal etc., and ignore the other factors highlighted in the previous section. To the best of our knowledge, a generalized multimedia search results discovery mechanism, particularly in aggregated search, is the first to address in this research. Notably, we provided a balanced architectural approach for information discovery, emphasizing the dataset, data-model, and information diversification equally. We used real-dataset retrieved from the search engines in real-time. We instantiated a non-linear graph data modal consisting of diverse information while preserving the semantics and similarity relationships. Finally, we provided a theoretical background to foster exploration and discovery activities.

Our component-based architecture design includes sub-components. Each sub-component produces a consumable output. There are five main components, referred to as (A) Search Results Aggregation; (B) Multimedia Document Creation; (C) Multimedia Documents Grouping; (D) Graph Instantiation; (E) Semantic Lookup List, components as illustrated in Fig. 2. Each component is concerned with delegated responsibility, and their internal working is separate from each other. A discussion on each component is provided in the following sections.

(A) Search results retrieval & aggregation

Search results retrieved from the search engines are presented in the form of disjoint verticals. Adversely, users’ information needs are becoming complex and multi-modal, requiring the employment of multimedia artifacts for satisfaction. To aggregate scattered disjoint verticals (P₁), we introduced a search results aggregation component. The aggregation process of this component is subdivided into three steps.

(i) Vertical retrieval

Definition: We define vertical as a specialized assembly of same-typed search results and retrieval as a process of obtaining them from some external source.

Formalization: Let S be the set of α, β, γ, λ respectively given as S = {α, β, γ, λ}, where α is defined as a set of video snippets V given as α = {{V^ψ₁, V^ψ₂, V^ψ₃, … , V^ψ_n}, {V^ϕ₁, V^ϕ₂, V^ϕ₃, … , V^ϕ_n}}, β is defined as a set of news snippets N given as β = {{N^ψ₁, N^ψ₂, N^ψ₃, … , N^ψ_n}, {N^ϕ₁, N^ϕ₂, N^ϕ₃, … , N^ϕ_n}}, γ is defined as a set of image snippets I given as $\gamma =\{\{I_1^{\psi },I_2^{\psi },I_3^{\psi },...,I_n^{\psi }\},\{I_1^{\mathrm{\Phi }},I_2^{\mathrm{\Phi }},I_3^{\mathrm{\Phi }},...,I_n^{\mathrm{\Phi }}\}\}$, and λ is defined as a set of web snippets W given as λ = {W^ψ₁, W^ψ₂, W^ψ₃, … , W^ψ_n}, where ψ and ϕ denotes the textual and visual modality associated with a snippet respectively.

Instantiation: We retrieved top hundred search results from each web, news, image, and video verticals. Since exploratory and discovery tasks require increased recall over precision (P₄), we chose to retrieve maximum search results from the API provider. With each search result, we preserve the metadata associate with it. The verticals are retrieved from the Google search engine in real-time because Google is highly preferred by web users (Ali & Gul, 2016). The Table 1 shows the vertical retrieval parameters. Figure 3 outlines the possible features of a snippet.

Table 1:

Parameters for verticals retrieval.

Vertical	# of results (n)	Source	Modality	Feature(s)
Web	≤100	Google	Textual	Title, Description, URL
Video	≤100	Google	Textual + Visual	Title, Description, URL,
				Thumbnail, Date
News	≤100	Google	Textual + Visual	Title, Description, URL,
				Thumbnail, Date
Image	≤100	Google	Textual + Visual	Title, URL, Thumbnail

DOI: 10.7717/peerj-cs.449/table-1

(ii) Verticals aggregation

Definition: We define verticals aggregation as a single container of all the retrieved disjoint verticals.

Formalization: Let X be the subset of S consisting of the all the elements α, β, γ and λ from S. We consider only textual modality information given as $X={\sum }_{i=1}^n{S}_i^{\psi }$.

Instantiation: Each retrieved snippet has unwanted data (e.g., HTML tags, numbers, special characters, etc.). These impurities do not add meaning to the semantic analysis. We are restraining to perform extra pre-processing steps such as stopwords removal and stemming. It results in the loss of contextual information necessary for semantic analysis. Afterward, we preserve the scattered disjoint verticals textual data inside a single container as a linear list.

Architectural implementation

We implemented our architecture in Python 3 programming language using publicly available libraries. Search results are retrieved using freely available APIs to fetch the verticals from a search engine in real-time. We used Google search engine to retrieve the search results belonging to the web, news, image, and video verticals. We preserved the metadata associated with each snippet, such as the URL, title, date, length, description, and thumbnail, where available. For text summarization, we used LexRank (https://gist.github.com/rodricios/fee45381356c8fb36004/) extractive text summarization algorithm. Semantic analysis is done using SBERT’s (https://pypi.org/project/sentence-transformers/) sentence embedding on pre-optimized bert − base − nli − mean − tokens (https://github.com/UKPLab/sentence-transformers/) pre-trained modal and agglomerative clustering using the ward’s linkage method from sklearn (https://scikit-learn.org/) python library to obtain the clusters. We used Networkx (https://networkx.github.io/) python library to instantiate an undirected network to build the graph. Each node represented either a web snippet, multimedia document, or a multimedia document cluster. The snippet nodes attribute includes their metadata. The multimedia document and multimedia document cluster nodes attribute include their summarized text. Figure 4 shows the visualization of the instantiated graph generated from Cytoscape (https://cytoscape.org/).

Results

There is still no standard empirical evaluation measures for evaluating the aggregated search approach effectiveness (Li et al., 2017). These approaches are mostly considered in terms of the achieved precision & recall (Rashid & Bhatti, 2017) and judgment reports from the human experts (Ruotsalo et al., 2018). Calculating precision and recall in our case is a non-trivial task. It is mainly due to the nature of the data. Therefore, we used a real dataset with no prior labeling by human experts. Our empirical evaluation measures mostly depend on metrics requiring no initial labeling of data. We used internal clustering stability measures to evaluate the internal cluster model stability (Wani & Riyaz, 2016), and clustering accuracy based on the judgment of the human experts (Ruotsalo et al., 2018). We obtained accuracy and stability scores by dispatching pre-defined queries on Google’s real dataset.

We collected queries from the recently published ORCAS (Craswell et al., 2020) dataset consisting of 10 million distinct records. Selecting all queries in the dataset for evaluation purposes was not practical. Hence, we performed bi-gram and tri-gram query analysis on the ORCAS dataset. Afterward, we selected 25 queries from the top 100 most repeating bi-gram and tri-gram combinations. The average query length for this evaluation was set to 2.5 words. The chosen length was due to a recent study in (Degbelo & Teka, 2019) indicating average user query length between 2.44 and 2.67 words, which confirms that users’ information needs are becoming exploratory. Since in exploratory search, user needs are ambiguous, and the primary objective is to gain an overview of the information. Users type short queries instead of well-articulated longer queries as in the lookup search scenarios (Athukorala et al., 2016). We selected queries covering broad aspects. Therefore, an average query length of 2.5 words was considered based on the average of 2.44 and 2.67 words.

Comparison & discussion

Our approach outperforms in terms of accuracy (99%) in comparison to the approach provided by Achsas & Nfaoui (2018) (89%). It mainly can be due to variations in the data used for model training, choice of deep learning model, and parameterization process. We have performed rigorous and statistically significant empirical evaluation using average scores from human experts having diverse backgrounds and internal clustering stability measures. It presents as a baseline, and a promising start for future search results aggregation approaches. This signifies that the previous researches in this domain are excessively concerned with innovations in the existing techniques, which we believe, is unnecessary while the present search engines are still suffering from the major issues discussed in this paper. We also made an effort to emphasize the fact that the existing techniques are sufficiently optimized to solve real issues and there exists a need for attention from the researchers towards more emerging trends.

Each research utilizes different techniques and mechanisms to provide information exploration and discovery. We have extracted the major parameters and their possible values for an in-depth comparison of our approach with existing state-of-the-art. To ease comprehension of these parameters, we have further categorized parameters according to their purpose, as displayed in the Table 5. The provided functionalities are marked with the “+” symbol, whereas the missing functionalities are left blank.

Table 5 emphasizes the three significant aspects of discovery techniques. The first aspect is searching for search results, including search type, search results granularity, and searching activity. The second aspect concerns data management, including information sources, instantiation of data-modal, and assembling mechanism. Finally, the third aspect is concerned with technical information retrieval aspects of the discovery and exploratory approaches, including media sources and information retrieval modal.

The most crucial factor in information discovery is flexibility in representing the information to avoid information overload. Most of the existing research solely relies on filtering capabilities but lacks in providing appropriate granularity control of the search results (di Sciascio, Sabol & Veas, 2016). Our approach provides three-level granularity; snippets, multimedia documents, and multimedia documents clusters. The data-modals employed by the existing researches are mostly centered around a specific domain and specific data. They mainly include the scientific domain having millions of literature as a dataset. Approaches providing real datasets were also primarily concerned with integrating a few verticals such as web and image (Koh et al., 2006). To enable our approach to be generic and applicable to all the domains and datasets, we presently use only real datasets to observe our approach’s integrity even in the most variate and uncertain data coming from the search engines in real-time.

Information exploration and discovery is a long, non-trivial, and non-linear journey. To foster non-linear navigation of the search results, existing literature mostly instantiated a graph data-modal using either existing domain knowledge, such as ontologies (Khalili et al., 2017; Lisena et al., 2017; Kanjanakuha, Janecek & Techawut, 2019), or using some generic similarity measures (Rashid & Bhatti, 2017). Our approach uses domain-independent semantics and similarity measures to construct a non-linear graph to provide non-linear means of search results exploration and discovery.

Information management is also an essential factor in enabling information discovery and a compelling exploration of the search results. Numerous information management approaches organize and present the users’ search results, increasing their cognitive abilities. These approaches include linear and non-linear browsing of information and summarization. However, previously, these were implemented as disjoint components, combined on a single interface (Fung & Thanadechteemapat, 2010). Our approach unifies all of the specific techniques and encapsulates it in a single component. With our generic information discovery architecture based on a strong theoretical background and promising empirical evaluation results, we hope to provide a new baseline for future researches on relational aggregated search and search engines alike.

Conclusion & future work

In this research, we proposed a generic discovery architecture using multimedia search engine results. A brief discussion on information exploration and theoretical discovery background was provided, and an architectural solution was formalized and instantiated. Before this work, the exploration and discovery of information on the web search engine were leveraged using traditional heuristics. We identified potential gaps and issues in the current general web search engine approach. To overcome these issues, we presented a new baseline using search results aggregation. Our approach was employed using state-of-the-art sentence embeddings. We bridged the gap between the abundant multimedia contents by encapsulating semantically multimedia artifacts in multimedia documents and summarizing them. Moreover, we eased the navigation problem in the search results space by grouping multimedia documents in semantically similar patches.

The proposed discovery architecture emphasizes all the aspects of the discovery, including information exploration and lookup. We supported information exploration by providing the nonlinear proximal navigation and exploration support through the instantiation of a complex graph and lookup searches through a semantically fully-blended ordered linear search results list. Finally, a comprehensive empirical evaluation was presented. The empirical evaluation out-performed previous aggregation approaches at all granularity levels of aggregation provided in this research. To the best of our knowledge, our approach is the first to be assessed comprehensively from the system and the user perspective on the dataset and the queries obtained from the user and the search engine, respectively, in real-time.

We believe that the user experience and the user interface both play an important role to influence information discovery. This aspect requires a detailed and comprehensive discussion, which was out of the scope of this research. In the future, we look forward to providing a more ablative evaluation from the usability perspective of architecture involving an even broader audience with extremely varied backgrounds and experiences with more focus on human aspects, including user interfaces. We have intentions to provide the adaptable clustering of multimedia documents by considering the users’ diverse information need and information-seeking behavior. We are interested in investigating multimodal verticals aggregation and exploiting various nonlinear data models consisting of multiple modalities in the enhanced discovery of aggregated multimedia based document search results in real-time scenarios.

Supplemental Information