Website reorganization using an ant colony system

Abstract

The growth of the Internet has led to many studies on adaptive websites based on web usage mining. Most studies focus on providing assistance to users rather than optimizing the website structure itself. A recent work pioneered the use of 0–1 programming models to optimally reorganize websites based on the cohesion among web pages obtained by web usage mining. The proposed models reduce the information overload and search depth for users surfing the web. A heuristic approach has also been proposed to reduce the required computation time. However, the heuristic approach involving two successive 0–1 programming models still requires a very long computation time to find the optimal solution, especially when the website contains many hyperlinks. To resolve the efficiency problem, this study proposes an ant colony system to reorganize website structures. The proposed algorithm is tested extensively with numerical examples. Additionally, an empirical study with a real-world website is conducted to verify the algorithm applicability.

Introduction

Designing and redesigning a complex website are increasingly challenging tasks for webmasters due to pervasiveness of the Internet. Perkowitz and Eizioni (1997) noted that different visitors, or the same visitor visiting at different times, may have different goals. Furthermore, a site may be designed for a particular purpose, but be used in unanticipated ways in practice. Moreover, many sites outgrow their original design, accumulating links and pages in unanticipated places. Web pages might become obsolete as time passes and the need for information changes. Web visitors then have to navigate redundant pages to reach their required information, prolonging the navigation time. Facilitating the browsing of a complex website is increasingly important in modern web management.

Perkowitz and Eizioni (1997) defined adaptive websites as sites that automatically improve their organization and presentation by learning from visitor access patterns. The browsing efficiency of a website can be enhanced by two basic directions, namely customization and transformation. Customization means aiding a user by providing additional meta-information or guidance. Although customization has many advocates, it has obvious limitations. First, additional index pages containing suggesting URLs might result in information overload for new visitors, or become useless for old visitors with different goals from those of previous visits. Second, web usage mining is computationally intensive for a large website, particularly one that is disorganized or has many obsolete pages. Additionally, obsolete pages raise the required disk space to store the pages, server logs, and user browsing history. Recent investigations have considered transformation, which substantially reorganizes the links among web pages. Transformation involves partially or completely changing the website structure to ease the navigation for a large set of users. The benefits of reorganizing the website structure include enhanced web usage mining efficiency and lower memory requirement for storing the pages, server logs and user browsing history.

Both customization and transformation depend on mining the web server logs to discover access patterns. Web mining involves adopting data mining techniques to discover useful information hidden in the web content and concealed user behaviors. Web mining is helpful in fields such as adaptive websites (Perkowitz & Eizioni, 1997), customized search engines (Madria, Bhowmick, Ng, & Lim, 1999) and customized e-commerce (Hollfelder, Oria, & Ozsu, 2000). Cooley, Mobasher, and Srivastava (1997) categorized web mining as web content mining and web usage mining. Web content mining digs knowledge from web content, for instance, to enhance search engines, while web usage mining digs user browsing patterns from web server logs to discover user needs, and thereby provide assistance to users. Web usage mining involves three main tasks, preprocessing, pattern discovery and pattern analysis (Srivastava, Cooley, Deshpande, & Tan, 2000). Preprocessing involves abstracting the usage, content and structural information from various data sources for pattern discovery. In usage preprocessing, the click-stream for each identified user is split into sessions. A 30-min timeout is often adopted as the default method of splitting a user’s click-stream into sessions. Pattern discovery adopts methods and algorithms developed for several fields such as statistical analysis, associate rules, clustering, classification and sequential patterns. The methods adopted depend on the purpose of mining. Pattern analysis filters out uninteresting rules or patterns from the set discovered in the pattern discovery phase.

Adaptive websites have become important applications for Web usage mining owing to the pioneering work of Perkowitz and Eizioni (1997). Most works on adaptive websites have focused on gathering related pages or users to provide assistance to the users. Users are clustered according to similar browsing patterns, whereas pages are clustered according to related content. In both applications, permanent or dynamic HTML pages can be created to recommend related hyperlinks to the user based on the user’s query or browsing history. However, clustering users appears to be more difficult than clustering web pages, possibly because user behaviors frequently change. Consequently, a user cannot be expected to have a consistent browsing behavior, and therefore cannot be categorized into a single group. Conversely, if a user can belong to many groups, and a user group is defined as users visiting the same page group, then clustering users becomes identical to clustering Web pages.

However, usage-based web page clustering can be performed in two ways: gathering similar sessions or transactions, and gathering Web pages with high co-occurrence frequency. In the first approach, a session is a group of web pages, similar to the basket items in association rule mining, and is therefore an object in the hyperspace to be clustered. Traditional distance-based clustering approaches can be used for session-based page clustering. Other clustering techniques, such as the adaptive resonance theory adopted by Kang and Cho (2001), the self-organizing map of Smith and Ng (2003) and ant colony clustering of Abraham and Ramos (2003), have also been applied to this clustering problem. However, as indicated by Mobasher, Cooley, and Srivastava (1999), session clusters are not effective for capturing an aggregated view of frequent user access patterns. Each session cluster may contain thousands of user sessions involving hundreds of URLs. Furthermore, determining the similarity between session vectors is not a trivial task. Hence, association rule mining techniques, such as the Apriori algorithm (Agrawal & Srikant, 1994) and the Association Rule Hypergraph Partitioning technique (Hollfelder et al., 2000, Kang and Cho, 2001) used by Mobasher et al. (1999), have been adopted to identify page pairs with high co-occurrence frequencies. Perkowitz and Eizioni (2000) processed access logs into visits, and obtained a similarity matrix containing co-occurrence frequencies between pages to represent a graph. Clusters in the graph were identified, ranked and presented to the webmaster. Finally, an index page consisting of links to the pages in each selected cluster was created.

Notably, none of the previous works had proposed reorganizing websites. Only dynamic or static index pages were provided to facilitate visitor searches, while the original hyperlink structure remained untouched. However, websites require reorganization, especially when they are old, or when their content outgrows the original structure. Fu, Shih, Creado, and Ju (2002) provided approaches to modify hyperlinks locally to evolve a website along with its usage. Hyperlinks are rearranged, and pages are merged or removed when necessary. Nevertheless, their method reorganizes websites locally rather than globally. Accumulating local refinements do not necessarily lead to global optimality.

Website structure reorganization can be considered as a special graph optimization problem. Lin (2006) recently pioneered the approach of adopting 0–1 programming models to formulate and solve problems based on the cohesion between web pages obtained from web usage mining. The proposed models consider reducing the information overload and search depth simultaneously for users surfing websites. Additionally, a heuristic approach involving two successive 0–1 programming models has been proposed to reduce the required computation time. However, due to the combinatorial nature of the models, the heuristic approach still requires a very long computation time to obtain the optimal solution, especially when the website contains many hyperlinks. To resolve the computation efficiency problem, this study presents an ant colony system to reorganize website structures. The next section introduces the website reorganization problem and Lin’s method (Lin, 2006). Section 3 presents an overview of ant colony optimization. Section 4 presents the proposed ant colony system, and Section 5 uses simulated data to compare the ant colony system approach with the 0–1 programming approach. Section 6 addresses the proposed method’s applicability to real-world problems by reorganizing a real-world website. Section 7 draws conclusions.