Multi-Layer Graph Generative Model Using AutoEncoder for Recommendation Systems

: Given the glut of information on the web, it is crucially important to have a system, which will parse the information appropriately and recommend users with relevant information, this class of systems is known as Recommendation Systems (RS) -it is one of the most extensively used systems on the web today. Recently, Deep Learning (DL) models are being used to generate recommendations, as it has shown state-of-the-art (SoTA) results in the field of Speech Recognition and Computer Vision in the last decade. However, the RS is a much harder problem, as the central variable in the recommendation system’s environment is the chaotic nature of the human’s purchasing/consuming behaviors and their interest. These user-item interactions cannot be fully represented in the Euclidean-Space, as it will trivialize the interaction and undermine the implicit interactions patterns. So to preserve the implicit as well as explicit interactions of user and items, we propose a new graph based recommendation framework. The fundamental idea behind this framework is not only to generate the recommendations in the unsupervised fashion but to learn the dynamics of the graph and predict the short and long term interest of the users. In this paper, we propose the first step, a heuristic multi-layer high-dimensional graph which preserves the implicit and explicit interactions between users and items using SoTA Deep Learning models such as AutoEncoders. To generate recommendation from this generated graph a new class of neural network architecture-Graph Neural Network-can be used.


Introduction
Information overload is one of the biggest caveats in online information systems, as an average visitor might be only interested in a tiny subset of this information. Recommendation systems (RS) are used to fetch these relevant subsets of information for every visitor from the huge catalog of options in real-time. The application of recommendation system is in almost every domain of web [Adomavicius and Tuzhilin (2005)], For example, E-Commerce, Online News Platforms, Personalized Advertisements, Social Networking, Academic citations, Search Engines, and Entertainment to name a few, For this study, we will be focusing on movies recommendations. The terms User and Item is used to represent the visitor and product/service respectively. Classically recommendation systems are classified as collaborative filtering, content-based, and hybrid recommendation systems 4 . Generally, a matrix R, called as utility matrix, is used to represent user-item preferences in a two-dimensional sparse matrix and the data is teased out of the matrix R along with additional information to calculate recommendations. In industry, the recommendation system is no longer just a tool to save user's time and improve user experience but also important for sales of the products/services. For example, it is reported that 80% of the movies watched on Netflix [Gomez-Uribe and Hunt (2015)] and 60% of the videos watched on Youtube [Davidson, Liebald, Liu et al. (2010)] came from recommendations. Due to the direct and active interest of both industry and academia, the rate of improvement in this research area is conspicuous. The latest improvement is the use of Deep Learning (DL) methods [Zhang, Yao and Sun (2017); Covington, Adams and Sargin (2016)]. The use of DL in the field of Computer Vision and Speech Recognition has shown tremendous success in the past few decades. However, we believe a new approachexpanding the classical RS approaches-is required to fully exploit the capabilities of DL models in the realm of recommendations. This paper is a step in that trajectory. RSs have been exceptionally improved in the last few years despite many challenges; the biggest challenge in our opinion is the Human factor, the chaotic relationship between the visitor and the information. Most of the contemporary classical RS approaches understate these chaotic variables in the system, as it is awfully difficult to fully quantize it in Euclidean space. The adoption of DL models in the RS has given us the opportunity to represent this chaotic relationship in a complex higher-dimensional space. In this research, the latent-representations learned from the utility matrix R, using Autoencoders, are embedded with the bipartite-graph, of R, to generate a high-dimensional multi-layer knowledge-graph which can be used later to generate recommendations for the user in an online manner. The proposed generative model will be discussed in section 3. The rest of the paper is organized as follows. Section 2 will discuss the related work, section 3 explains the proposed method. Section 4 describes the experiment. Section 5 concludes and discusses the future work.

Related work 2.1 Graph-based recommendation
Graph-based knowledge representation has been studied for decades and it has emerged as an important model for studying complex multi-relational data, the buzzword termed by Google 5 , Knowledge Graph [Ehrlinger and Wöß (2016)] is mostly used in industry and academia to refer this model. This knowledge bases stores data in RDF triples (subject, predicate, object). Amazon is exploring the possibilities of building a knowledge-graph inspired product graph 6 [Dong (2018)]. The graph-based approaches has been extensively researched as a link prediction problem for the recommendation systems in the literature [Chang, Harper and Terveen (2015) (2018)]-In this models the User-Item and their relation is represented as a bipartite graph, the prediction of the missing rating is achieved by the link prediction approach. Graph-kernel based link prediction in bipartite graphs [Li and Chen (2013)], Graph-based Clustering [Reddy, Kitsuregawa, Sreekanth et al. (2002)], and Pinterest has implemented a bipartite graph based recommendation on its platform with 3+ Billions of items and 200+ Millions of user in real time.

Autoencoder based recommendation
Autoencoder is a type of artificial neural network which reconstructs the input in the output layer. It has two major components: Encoder and Decoder. In the intermediate layer (Bottleneck Layer), z in the Fig. 1 is known as latent features vector-Encoded lower dimensional representation of the input vector-and it serves as an input to the decoder. There are many variants of autoencoders and most of them can be used for recommendation task [Zhang, Yao and Sun (2017)].

Methodology
The original data-with m users, n items and a partially observed utility matrix R ∈ ℝ m * n -is represented as a Bipartite-Graph with three entities, where U and I as the sets of nodes and elements of R as the weighted edges rij connecting U and I, as shown in Fig. 2. Each user can be represented by a partially observed vector r (u) = (Ru1,…,Rum), similarly for each item r (i) =(Ri1,…,Rin), where users u ∈ U=1,…, m and items i ∈ I=1,…,n from R, as shown in Fig. 3. The sets of vectors r (u) and r (i) are used as the input to autoencoders, to learn the latent features. Latent features vector-bottle-neck layer-learned from the vectors r (i) can be represented as the latent features representations-the top layer of Figs. 3 and 4of the users u ∈ U in respect to the items i ∈ I, where the weights from the bottle-neck layer to the output layer are used as the weighted edges between the features representation layer to the users layer-top two layers in the Fig. 4. Similarly, the latent features vectors of the vectors r (u) are represented in the graph-bottom two layers in Fig. 4. The Fig. 3 illustrates the proposed model. It can be noticed that the edges (dotted lines) between the learned feature layer and the entities layer are denser than the original (solid lines) bipartite-graph, as the autoencoder predicted the missing ratings. For this research, we have used only rating between useritem but additional information can be injected to make the generated graph more robust, our future efforts are in those directions.

Conclusion and Future Work
In this paper, we proposed a novel multi-layer graph based knowledge-representation generative model-using Autoencoders. The model learns the latent-features from the utility matrix R-a sparse user-item preference matrix-then the learned latent-features are embedded with the original information to generate a high-dimensional knowledge graph, which can be later used to generate a high quality recommendation in an online manner. In the future work, we will try to explore the possibilities of generating a finer graph which will capture all the supplementary information of the user and item using state-of-the-art embedding techniques and ways to efficiently use that graph to generate recommendations. The use of other variants of Deep Learning architectures is also a rational possibility.