1. Introduction

Web Impact Assessment studies are gaining more attention day by day since it is useful to improve the organization’s website content for attracting the customers through their website. This type of Web impact studies were done mostly in academic institutions and government department’s websites. Currently, private organizations are also showing more interest in improving their website through Web Impact Assessment studies.

Larson, (1996)[1] initiated the Bibliometric study of World Wide Web and it was followed by a Spanish scientist Rodríguez i Gairín, J.M (1997)[2]. Ingwersen (1998)[3] developed a quantitative method to measure the average web links called the Web Impact Factor (WIF). WIF calculated by the number of pages with links to the unit was divided by the number of pages published on the Web. Björneborn and Ingwersen (2004)[4] were defined Webometrics within the framework of Informetric studies and Bibliometrics, as belonging to library and information science, and as associated with Cybermetrics as a generic subfield.

According to Noruzi (2006)[5] the Web Impact Factor (WIF) is not the perfect tool and believed that more sophisticated facilities will come in future to measure the website quality. Thelwall (2009)[6] defined the term Webometrics that “the study of web-based content with primarily quantitative methods for social science research goals using techniques that are not specific to one field of study”. The tools such as Search Engines, Web crawlers, Webometric software etc., that are applied for data collection from the web are called Webometric Tools.

Most of the well-known search engines offer special keywords to search for matches only in web elements such as page title, domain, host, hyperlinks etc. . These keywords can be combined with other keywords by using Boolean operators to narrow down the search. Advanced link search query options were available in all popular search engines to obtain hyperlink counts. The search engines like “Altavista”, “AllThe Web”, “Bing” ,“google” ,“hotbot”, “yahoo”, etc, (Ingwersen, 1998[3]; Smith, 1999[7], Smith and Thelwall, 2002[8]; Bar-ilan, 2004[9]) are user-friendly.

This paper reviews how this Web Impact Assessment data has been collected using various Webometrics tools like search engines, web crawler etc., and concludes with the analysis of the status of Webometrics tools as on date.

2. Search Engines

Web search engines like Google, Yahoo! and, Bing can give researchers free access to huge amounts of data about the content and link structure on the Web. Aguillo (2000)[10] evaluated a client-side based, low cost (shareware) programs to collect data, tracing, indexing, analysis and visualization for World Wide Web medical resources. Web search engines collect data similar to Web crawlers that researchers use to collect link data. In fact, Web search engines have three different parts: the crawler, the indexer and the search engine interface where the queries are inserted (Bar-ilan, 2004[9]). Aguillo et al. (2006)[11] used advanced features of search engines to collect data from Web for university rankings.

2.1. Altavista (http://www.altavista.com)

Altavista was the best search engine for web impact Assessment studies. Many researchers have used this search engine since it has advanced features.

Thelwall (2001)[12] used Alta Vista Search engine to compare the sources of links for academic web impact factor calculations. He found that the WIF was less able to differentiate in more homogeneous subsets of universities, although positive results are still possible. Smith and Thelwall (2002)[13] collected data for interlinking between Asia-Pacific university web sites. They concluded that the nature of larger web sites covered was qualitatively different from that of smaller ones, making the deduction of relationships between the hosting institutions difficult from the link counts alone.

Noruzi (2005)[14] studied time series of WIF for the University of Tehran in order to monitor AltaVista search engine performance and showed that there existed large increases and decreases during the time. Agarin and Nwagwu (2006)[15] studied the exploratory analysis of Nigerian universities’ website links. They found that Nigerian universities are not linking themselves and observed no direct links existed between any two universities websites.

Elgohary (2008)[16] studied the Webometric analyses of Arab Universities on the web and found Web Impact Factors of Arab universities as well as the relation to other variables such as language. The paper provided some measures to the universities and how they were viewed by the outside world. Asadi and Shekofteh (2008)[17] evaluated the relationship between the research activity of Iranian medical universities and their Web Impact Factor. Their results showed that WIF index may not be an appropriate index for medical university website validation. It appears that the number of in-links is a better index for evaluation.

Jalal et al (2009a)[18] studied Webometric analyses of web presence of selected Asian Countries. They attempted to bring out the importance of Webometric research and tried to throw some light on a few aspects, specifically hyperlink studies so as to reflect the present status of Asian countries and their relative position among themselves.

Jeyashankar and Ramesh Babu (2009)[19] examined the Webometric study of Tamilnadu Universities. Their study revealed that some universities in Tamil Nadu had higher number of web pages, but correspondingly their link pages were very small in number and their websites fell behind in their simple, self link and external link web impact factor.

Islam and Alam (2011)[20] studied and analysed websites of private universities in Bangladesh. Their study revealed that some private universities in Bangladesh have higher number of web pages, but their link pages are very small in number, thus the websites fell behind in their Overall WIF, self link, external links and Absolute WIF. Finally, it was shown that these universities did not have much impact factor on the web and were not known internationally.

Adekannbi (2011)[21] carried out a comparative study between African and World Universities on web link analysis data which were extracted using the AltaVista search engine. This research has intensified the need for further investigation into the motivation for web connectivity between African universities and the top world universities and the means by which such relationship can be used to improve academic activities in African universities. Thanuskodi (2011)[22] studied of Webometric Analysis of Private Engineering College Websites in Tamil Nadu. He reported that the websites is updated only in few engineering colleges remaining websites did not mentions time or date in the homepage. He further stated that webometric techniques are still in experimental stage in testing whether the classical bibliometric methods applied to the web are reliable and feasible means of comparing the analysis of websites. Vijayakumar et al (2012)[23] studied identification of web presence and their links among SAARC countries. His research revealed that India occupied top position among the SAARC countries.

Yahoo took over the AltaVista search and withdrawn its ‘linkdomain’ command from 2011.

2.2. Yahoo (http://www.yahoo.com)

Yahoo is one of the popular search engines used as Webometric tool for web impact factor applications (Danesh et al,2008[24] ; Jalal et al, 2009[25]; Jati,2011[26]; Yu and Lian,2011[27]). Danesh et al (2008)[24] studied the hyperlink analysis of the websites of Iranian Ministries during 2008. = They concluded that the final success of a website is dependent on factors such as quality, size, language, history, content and some other factors and that one or two restricted factors cannot be declared as sole reasons for its success.

Jalal et al (2009)[25] studied Webometric analysis of central universities in India. They made an attempt to rank the Central Universities in India using appropriate Webometric indicators. Their results revealed that University of Delhi had the top rank while Sikkim University occupied the last among Central Universities in India. Jati (2011)[26] studied the impact, visibility, rich content and influence of Indonesian university websites. He concluded that the University of Sriwijaya had the highest number of webpages and was first ranked with respect to number of webpages. Universitas Bina Nusantara had the highest impact on web with a far greater number of inlinks as compared to the number of webpages on its website.

Yu and Lian(2011)[27] studied the Link Analysis of China’s Governmental Tourism Websites. They reported that web page counts, external backlink counts and total backlink counts can be better evaluation indices for governmental tourism websites of Chinese provincial capitals than WIFs. Many authors preferred to use Yahoo search engine since it had “linkdomain” command to count backlinks. Altavista were taken over by Yahoo and ceased its ability to gather hyperlink counts. (Thelwall and Sud. 2012)[28]

2.3. Google (http://www.google.com)

Google has an advanced search facility, but does not support the same level of Boolean querying as AltaVista. Although Google is the most used search engine at the moment and it is not recommended for collecting link data for link analysis purposes. (Li, 2003)[29]

Bar-ilan (2004)[30] studied the usage of hypertext links by the 8 Israel universities and found that links between universities are inserted as ‘Web Convention’. Lu, et. al (2011)[31] used Google to investigate the Web Resource Distribution in the field of Information Science and stated that the difference between governmental and commercial domains and between educational and organizational domains are not significant.

Presently Google allow ‘site’ command for its users. A search for site: www.xxx.xxx returns URLs that begin with www and a search for site.com returns URLs including number for all subdomains.

2.4. Exalead (http://www.exalead.com)

Boell et al (2008)[32] used Exalead search engine for a Webometric Analysis of Australian Universities using Staff and Size dependent Web Impact Factors. Their study showed that size dependent WIF values declined for most Australian universities over the last ten years, while staff dependent WIFs rose.

Presently Exalead permits ‘domain’ command for its users. But it yields fewer number than the Google and Bing’s ‘site’ command.

3. Web Crawlers

Web crawlers are programs that are programmed to start from a given Web page and to use links from it to move automatically and independently on the Web from one page to another, from one website to another, collecting and saving information about the links and the content on the websites, until there are no more links to follow. Web search engines also use Web crawlers to find new pages and new sites to be added to their databases (Holmberg,2009)[33]. Web crawlers can be programmed to search only pages covering certain topics, to avoid certain pages and areas of the Web and to check for and exclude duplicate content (Thelwall, 2004)[34].

3.1. Link Discoverer

Yang and Qin (2008)[35] used LinkDiscoverer to evaluate its performance in Link analysis. The results showed that the LinkDiscoverer’s functions can well satisfy the needs for link analysis. Yang and Sun (2012)[36] used the same crawler to explore the link-based knowledge map in academic web space. There is not much information about the performance and special features available at present.

3.2. SocSciBot

SocSciBot is a specially designed, user friendly website crawler that is available free to use for academic research purposes. Duplicates were removed through comparison method of the HTML pages.

Onyancha and Ocholla (2006)[37] used SocSciBot in his study on web presence and impact of South African universities. Their study concluded that South African Universities provided a sufficient platform for Webometric evaluation in addition to providing an agenda for further research involving web-related developments of African universities.

Onyancha and Ocholla (2007)[38] used SocSciBot to link analysis study to compare Kenyan and South African Universities. This study revealed that Kenyan universities websites are in the initial stages of construction, but that South African universities have made remarkable progress in their web presence, which is at an advanced stage of development, equalling counterparts in more developed countries.

Thelwall (2010)[39] communicated to Mahmoud that SocSciBot crawls web sites and extracts the hyperlinks from them. SocSciBot will take a long time to crawl these web sites if there are many (more than 10) websites.

SocSciBot (http://socscibot.wlv.ac.uk/) is freely downloadable web crawler for link analysis research on a single web site or collection of sites.

3.3. Webometric Analyst/ LexiURL Searcher

Webometric Analyst is free software(http://lexiurl.wlv.ac.uk/) for academic use intended to conduct automatic web analyses for social science research. This software will create network diagrams, estimate the online impact of groups of websites. It will also retrieve information on a large scale about blogs, YouTube videos etc. The software possesses a range of inbuilt functions that are complete and self-contained in addition to a set of powerful and flexible commands that can be combined for different purposes.

Thelwall (2010)[39] communicated to Mahmoud that LexiURL Searcher gets data on links from Yahoo! and, can be used with the Network Diagram wizard option to get a network diagram of the links. This is faster than SocSciBot, but gives less complete lists of links between the websites. He recommended that research can start with LexiURL Searcher (Webometric Analyst) and switch to SocSciBot only if they really need the extra information that it gives. This will save a lot of time.

Thamm and Mayr (2011)[40] used LexiUrl Webometric tool to compare the Webometric with web-independent rankings of German Universities. They found that link impact could not be easily seen as a prestige factor for universities. Thelwall et al., (2012)[41] used this software to compare the methods for collecting web citation data for academic organisations. They compared link counts to two alternative methods: URL citations and organisation title mentions. Webometric Analyst has a facility to automatically combine the results of multiple searches and to eliminate duplicates if URL or domain counting is used..

Thelwall, and Sud (2012)[28] informed that researchers still depend on Commercial search engine’s API for collecting raw data for their Webometric studies.. These APIs allow automated data collection by letting programmers write a code to access search engine results. Yahoo closed its free search API, Google’s API had restricted it’s access from 2011 and Bing’s API 2.0 had limited it’s access for free from 2012 as an important data source for Webometrics and large-scale quantitative research.

4. Critical Analysis (Discussion)

Search engines play a critical role in Web Impact Assessment studies. No search engine is available to collect the data from Whole of the Web. The capabilities of Search engines were studied a number of times by many authors. Rousseau (1997)[42] expected that the Internet are volatile, and it cannot provide the same results and suggested that it is necessary to repeat the measurements at different times to evaluate the volatility of the Internet. Bar-ilan (2000)[43] compared the stability of results between Hotbot and Snap, and concluded that Snap is better than Hotbot. Thelwall (2000)[44] conducted a survey to test the Altavista reliability and found that it gives misleading calculations. .Web possesses the dynamic environment and it gives fluctuation in results. These differences may be due to search engines focus on text and content based retrieval algorithms. Hence, it is suggested to study the search engine’s performance over a period of time.

Ingwerson and Smith used Altavista for Web Impact Factor studies by using ‘link’ and ‘domain’ commands, and it was followed by many authors with universities/Higher academic institutions of their own country. Altavista and Yahoo are the most popular search engines used for Web Impact Assessment studies. ‘Link’ and ‘Linkdomain’ are the important commands used to collect the data for Web Impact Factor studies. Altavista was taken over by Yahoo and ‘linkdomain’ command was withdrawn from 2011. Google, Bing and Yahoo allow ‘site’ commands for their users. A search for site:www.xxx.xxx returns URLs that begin with www and a search for site.com returns URLs for all subdomains. Google and Bing gave the number of URLs and yahoo did not provide the number of URLs in the results. Since ‘linkdomain’ or equivalent command is not available, we can study only the web impact assessment with the presently available search engines.

‘SocSciBot’ and ‘Webometric Analyst’ are available for free to be used for academic purposes. It crawls one or more web sites and analyses them to create standard statistics about their interlinking and network diagrams of the interlinking. These two softwares extract quantitative data from web and use Application Programming Interface (API) to pull out all returns URLs (subdomains) directly to the hard disk will be useful for further data analysis. Web crawlers are depend on search engine’s API for Web Impact Assessment studies. Yahoo closed its free search, Google’s had restricted and Bing’s limited its API for free from 2012. (Thelwall and Sud, 2012)[28]

5. Conclusions

Search Engines and Web crawlers are used to collect the data for Web Impact Assessment studies. Web crawlers are better tools than search engines for Webometric research. Google, Yahoo & Bing are the search engines and Soscibot & Webometric Analyst are crawlers commonly used in Web Impact Assessment studies. . Since ‘Webometric Analyst’ utilised Bing’s API limited free search facility and possesses a range of inbuilt functions that are complete and self-contained in addition to a set of powerful and flexible commands that can be combined for different purposes. , So, it will be concluded that Webometric Analyst is the only best option available today for Web Impact Assessment Studies.

ACKNOWLEDGEMENTS

We would like to thank Dr. K. Saravanan and Mrs. Urmila Chandrasekaran for their helpful comments on the draft of this paper.