Research performance of marketing academics and departments an international comparison

We report the results of an analysis of the research impact of marketing academics using citation metrics for 2263 academics in the top 500 research universities in the Academic Ranking of World Universities based in Australia and New Zealand, Canada, the United Kingdom and the USA. The metrics are computed for publications from 2001 to 2013, which were collected in 2014 and 2015. We also report the same metrics for all universities in Australia and New Zealand that employ more than 4 marketing academics. The results provide an objective measure of research impact and provide benchmarks that can be used by governments, universities and individual academics to compare research impact. In an appendix we rank the top 100 university marketing departments in the top 500. © 2015 The Authors. Published by Elsevier Ltd on behalf of Australian and New Zealand Marketing Academy. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).


Introduction
Research is one of the primary functions of any university, but assessing research and researchers is a vexed issue.Despite this, we live in an environment in which there is an increasing focus on assessing research performance (e.g. the Excellence of Research Achievement (ERA) in Australia, the Performance Base Funding system in New Zealand and the Research Assessment Exercise in the UK).In making these types of assessments the research metrics used matter.There is an old saying that you cannot manage what you cannot measure.This logic underlies much of the recent effort to assess researchers and to allocate research funds.While there is some truth in this statement, there is a downside because, once you measure something, many people try to manage the measures and this can lead to distortions and misleading information.
There are two major research performance dimensions -quality and impact."Quality" refers to the degree of scholarship, which includes the significance and novelty of the contribution to knowledge, the complexity of the research problem addressed and the sophistication, complexity and novelty of the research methods."Impact" refers to academic impact, its use and acceptance by other researchers which, it is argued, indicates the progress of science.There are other dimensions to impact (e.g.impact on society and business), but these are not considered here.An often-used proxy for quality is the prestige of the journals in which a paper is published, as reflected, for example, in the Australian Council of Business Dean's rankings (www.abdc.edu.au),although it is worth noting the caveat the ABDC puts on such an approach to evaluating quality, as they comment "journal lists should be a starting point only for assessing publication quality and should not constrain researchers to a particular domain.There is no substitute for assessing individual articles on a case-by-case basis" (http://www.abdc.edu.au/pages/abdc-journal-quality-list-2013.html).Some other indicators of an individual researcher's quality include membership of learned societies, the university from which they gained their doctoral degree, research awards, research paper awards, invited papers and invited keynote addresses.
Impact can be assessed holistically and qualitatively (e.g. as occurred in marketing in the last research quality assessment exercise in Australia).However, it is more commonly measured in terms of citation metrics, as it was for most of the science disciplines in the Excellence in Research Achievement (ERA) assessment.Indeed, citations are often suggested as the "gold standard" of scientific impact and used as a proxy for quality because quality and impact are linked.For example the ERA justifies the use of citation metrics, arguing: "the more frequently an article is cited the more it is contributing to the stock of knowledge" and"Citations generally provide similar results to traditional peer review processes and can serve as a proxy" (arc.gov.au/media/arc_presentations_archive.htm#2013).
A substantial literature exists about the validity (e.g.MacRoberts andMacRoberts, 1989), reliability (e.g. van Raan, 2005) and value (e.g.Bornmann and Daniel, 2008) of citation metrics and at what level of aggregation these metrics can be appropriately used (Seglen, 1997).While high quality papers or researchers are likely to have high impact, this is not always the case.Particular papers and researchers who do well on research quality measures (i.e. they have published many papers with a high percentage of them in A* and A rated journals) may not do as well in terms of citations.This can occur because it takes time for a contribution to be appreciated and because the citability of a paper depends in part on the number of active researchers working in the subject area and their citation behaviour (Li et al., 2015).As a result, research in specialised topic areas may have little chance of getting a high number of cites, such as macro marketing, historical studies, esoteric research methods.
Citation metrics have been criticised as measures of research quality because they can be inflated for the wrong reasons, such as when they are cited for their errors, because there are "rubbish citations" from low quality non-academic sources, or they are self-citations (Bornmann and Daniel, 2008;Smith, 1981).However, research into citation patterns indicates that these issues have little impact on summary citation metrics such as the h-index or the g-index (Harzing, 2010).A further issue is that of journals gaming the citation metrics by editors asking researchers to include more citations to papers published in their journal before a paper is published.This strategy can enhance the citation metrics of some journals and partially explain their rapid rise in perceived status (Seglen, 1997).However, this raises a somewhat different set of issues than the citation analysis of particular articles, authors or departments and is not the focus here.
Here, we use citation metrics to compare the research performance of marketing academics and marketing departments in the top 500 universities in the Academic Ranking of World Universities that are based in Australia and New Zealand, Canada, the United Kingdom and the USA.We also report the same citation metrics for all Australian and New Zealand universities.Our research extends previous studies of the performance of marketing academics in Australia and New Zealand (Razzaque and Wilkinson, 2007;Soutar, 2013).This paper is based on and extends the results presented at the 2014 Australia New Zealand Marketing (ANZMAC) Conference, which compared measures of the research performance of marketing academics in the top 500 universities in the world based in Australia and New Zealand, Canada, the United Kingdom and the USA.Here we include additional data, using the same citation metrics, for all universities in Australia and New Zealand, rather than just those in the top 500 research universities.This provides a more complete picture of the research performance of ANZ universities and how they compare to the benchmarks established for the top 500 universities.In addition, in an appendix we provide a ranking of the top 100 marketing departments in the top 500.
In the next section we describe the methodology used, followed by a description of the results for individuals and departments based on academic rank.We then describe the same citation metrics for all Australian and New Zealand universities, before offering some final comments about the role such metrics can play.

Methodology
Google Scholar (GS) citation metrics are used to measure the academic impact of individual researchers and marketing departments because GS "generally results in a more comprehensive coverage in the area of management (including marketing)" (Harzing and van der Wal, 2008, p. 72).Further, GS is publically available and includes more journals than Thomson's ISI or Scopus and it also includes non-English language journals (Harzing, 2010;Meho and Yang, 2007).
In order to compare like with like, we included academics employed in research intensive universities, which were defined as those institutions in the Academic Ranking of World Universities (ARWU) in 2013 in which marketing was taught.We included those listed on university websites in May 2014.This resulted in a sample of 2263 marketing academics from 195 universities (123 from the USA, 27 from the UK, 22 from Canada and 23 from Australia and New Zealand).It is important to recognise that this means the benchmarks are likely to be higher than would have been the case if a random sample of all universities from these five countries had been included in the sample.Subsequently, we report the same citation metrics for all universities in Australia and New Zealand in which marketing is taught.
Following Soutar (2013), we focused on papers published this century (i.e. from 2001 to 2013) which served to standardise the comparisons and reduce age effects.Text books were excluded as they are often new editions of old books, making it hard to evaluate their real impact.However, research books, book chapters, journal articles and conference papers were included.We computed three citation metrics (the h-index, the g-index and the hg-index).The h-index is the number of papers that have at least that number of cites, so that an h-index of 10 indicates an author has published 10 papers with 10 or more citations.The g-index is the rank number of articles g for which there are g 2 number of citations.The minimum value of g is therefore h, which occurs when there are exactly h 2 total cites to the h articles.Rousseau (2006, p. 4) points out that these two indices measure different things but, "taken together, present a concise picture of a scientist's achievements in terms of publications and citations."Taking this viewpoint Alfonso et al. (2010, pp. 394-5) developed the hg-index, which is the geometric average of these two metrics.They argue this index fused "the benefits of both previous measures (while minimising) the drawback that each one of them presented, as the hg-index softens the influence of a high g-index when the h-index is low."They also noted some additional benefits of the hg-index, namely: • "It is very simple to compute once the h-and g-indices have been obtained.
• It provides more granularity than the h-and g-indices.This is especially interesting when compared with the h-index.As we have previously mentioned, to increase the h-index is difficult (more when the h-index is high) and it is usual to find that many different researchers have the same h-index with a very different number of total publications and cites.The hg-index provides a more fine-grained way to compare scientists.
• The hg-index is valued on the same scale as both the h-and g-indices (both represent the number of papers that comply with a condition about their cites).Thus, the hg-index is easy to understand and to compare with those existing indices.
• It takes into account the cites of the highly cited papers (the h-index is insensitive to highly cited papers) but it significantly reduces the impact of single very highly cited papers (a drawback of the g-index), thus achieving a better balance between the impact of the majority of the best papers of the author and very highly cited ones" (Alfonso et al., 2010, p. 395).
The hg-index is the most appropriate index to use, as it takes into account both the number of publications in the h-index and the number of citations (Moussa and Touzani, 2010).In addition, Soutar (2013) used this metric in computing benchmarks for Australian marketing academics and so the results can be compared.
The citation metrics were obtained using the Publish or Perish software (www.harzing.com/pop.htm) in April to May 2014 for the universities in the top 500 and in January 2015 for the ANZ universities.In each case the results were carefully checked to ensure the citations related to papers published by the marketing academic rather than to other researchers with similar names.The journal in which the article was published was used as a guide and use was also made of publications listed on a person's individual GS citation indices and on their university website.We believe measurement errors are small and are likely to bias upward if at all.

The results
As expected, the distribution of the hg-index was highly skewed, as shown in Fig. 1.For the marketing academics in the five countries the index ranged from zero to 77.The overall mean was 11.07 and the median was 9.17.Prior research suggests research productivity and impact is skewed (Soutar, 2013).Indeed, a General Pareto distribution fitted the data well, suggesting it is appropriate to look at percentiles when considering benchmarks.There were also differences between the hg-indices of more junior academic staff and more senior academics which is to be expected.
In order to make international comparisons it was necessary to classify academics into similar groups.As the USA was by far the largest group, it was decided to compare three academic levels based on their academic levels (i.e.Full Professor, Associate Professor and Assistant Professor).Australian, New Zealand and United Kingdom academic Level D (Associate Professor) and Level E (Professor) were included in the Full Professor category, Level C (Senior Lecturer) academics were classified as Associate Professors and Level B (Lecturer) academics were classified as Assistant Professors.Reclassifying level D academics as Associate Professors did not significantly change the results.
Significant differences existed in the mean hg-index across academic ranks (F = 466.01;p < 0.001).Full Professors had more impact (17.20) than did Associate Professors (10.08), who had more impact than did Assistant Professors (5.01).Scheffe's test confirmed that the three levels were all statistically different from each other.Benchmarks were therefore developed for each level, as well as for the sample as a whole.These benchmarks, which can be seen in Table 1, provide useful information for evaluating marketing academics in Australia and New Zealand.For example: 1. Any Assistant Professor (level B academic) who has an hgindex of 10 or higher is in the top 10% of such academics for these five countries (in top-rated Universities), but would also be in the top 50% of Associate Professors (level C academics).There are two such academics in Australia and New Zealand, who are both employed at different universities.2. Any Associate Professor (level C academic) who has an hgindex of 15 or higher is in the top 25% of such academics, but would also be in the top 50% of Full Professors (level D and E academics).There are five such academics in Australia and New Zealand, who are all employed at different universities.3.There are four Level E academics with hg-indexes of 39 or higher, putting them in the top 5% of Full Professors, who are employed at three different universities.A further six Level E academics have hg-indexes of 31 or higher, putting them in the top 10% of Full Professors, who are all employed at different universities.
The impact of individual university marketing departments was also examined.We computed the average hg-index for full professors of marketing.We did this because the main impact and reputation of a department tends to be based around its senior academics and because we did not want to disadvantage Universities with large departments with many junior academics.Table 2 shows the mean hg-index scores by country and overall.To ensure reasonable numbers in the averages, only departments with four or more Full Professors were included.The averages were virtually the  same for all countries and a Kruskal-Wallis test, which was used because of the small numbers of universities in all but the USA, found no significant differences (p = 0.82).This suggests senior academics in each country have very similar impacts.There was one university from Australia and New Zealand in the top 10%, another two in the top 25% and another two above the median, which is about what would have been expected given the number of universities from the various countries.A listing of the top 100 universities in rank order, based on the mean hg-index of their Full Professors is given in the Appendix.As noted earlier, only universities with 4 or more Full Professors are included in this list.So as to include all academics in assessing a department's research, and in the spirit of the Net Promoter Score (Reichheld, 2003), a Net Impact Score was computed for each department.This score was computed by subtracting the percentage of academics who had an impact of the median or less for their academic level from the percentage of academics who had an impact of the eightieth percentile or more for their academic level, ignoring those academics with impacts in between these two groups.A higher score implies a department had greater impact.A further non-adjusted Net Impact Score was also computed as a measure of "overall" impact.To ensure there were reasonable numbers in computing the scores, only departments with ten or more academics were included in the analysis.Table 3 shows the mean adjusted and non-adjusted Net Impact Scores by country.In this case, ANZ universities seems to lag, suggesting the "tail" of junior academics is longer in these two countries.This view was supported by the Kruskal-Wallis test as both were significant beyond the 5% level in this case.

Citation metrics for all universities in Australia and New Zealand
The previous results focused on universities in the top 500.To permit a more comprehensive description of the performance of ANZ universities, we describe in this section, the same kind of citation metrics for these universities.This required the collection of data for marketing academics employed in those universities in the two countries that were not in the ARWU 2013 rankings.As a consequence, 609 marketing academics in 45 ANZ universities were included in this phase, compared to the 385 marketing academics in 23 universities who were included in the international comparison.The data were collected in January 2015 and, for this analysis, the metrics were updated for ANZ universities included in the top 500.This was done because of the potential upward bias resulting from collecting the metric data approximately one year later.
While these additional data were used to assess all of the ANZ universities, the benchmarks reported earlier were not altered, so all ANZ universities were compared against these results.As was expected given the additional universities included, the mean hgindex scores for the marketing academics in the ANZ universities were significantly lower (t = 5.86).However, many ANZ academics were in the upper impact levels, as can be seen in Table 4. Interestingly, the four academics above the 99th percentile came from four different universities, two of which were not in any of the top 500 universities.
Table 5 shows the mean hg-index for the 35 ANZ universities with more than 5 marketing academics broken down by the three academic ranks defined earlier.Universities with fewer marketing academics were excluded not only because they have few marketing academics but also because they are likely to be more teaching oriented and this is also indicated by low citation metrics.We do not report metrics for ranks with fewer than 4 academics to focus on more meaningful department level comparisons.Comparing the results for Universities with 4 or more Full Professors, as defined earlier, 15 (75%) of the 20 ANZ universities with 4 or more Full Professors had average hg-index scores above the mean and median for the top 500 universities reported in Table 2, while 3 (6%) would be placed in the top 25% and 2 (4%) in the top 10%.
These are encouraging results that suggest ANZ universities are performing well, with 75% of these universities scoring above the mean and median.It is to be expected that a smaller percent of ANZ universities would be classified in the top 25% and 10% because comparisons are being made with international benchmarks established based on universities in the top 500.These results support the earlier suggestion that ANZ's senior academics' impact is in line with, if not above, world standards.Although it should be noted that there is a slight upward bias in these ANZ metrics because they were gathered some months later than the original data set.

Discussion and conclusions
These results provide some insight into the relative research performance of marketing scholars in the top 500 research universities in Australia, New Zealand, USA, UK and Canada and in ANZ universities in general.These are the nations in which much research in marketing is conducted and published.While the number of universities in the USA means they dominate the results, comparison across countries suggest there are no significant differences in the proportion of universities in the top tiers in each country.Other countries have marketing researchers and departments that are clearly world class.Australia and New Zealand fared well, as there was no difference in "full professors'" impact, although there was a longer tail of junior academics and 75% of all ANZ universities have above average scores compared to those in the top 500.
These kinds of data can and should inform government assessments of universities and can also aid universities and academics in judging their research performance.However, some potential limitations of the research should be recognised.First the data were collected in 2014 and 2015 and citation metrics are constantly growing, so the benchmarks described here can only be a guide.Our analysis of the top 500 universities excluded universities from many countries.But, most academic research in marketing does come from researchers based in these countries.We will report on the results from other countries in a subsequent publication.Another issue is the reliability of the h-index and g-index computed for each researcher.As was noted earlier, we made considerable effort to clean the data to ensure the publications included related to that person and not others with similar names.It should be noted that the tail of publications beyond the h-index does not affect the metrics and there is no reason to believe there are any systematic biases.But there is always some measurement error and, hence, small difference in scores should not be relied on.In subsequent work we plan to examine this issue more closely.
We have not reported individual metrics or names because we do not wish to personalise which is a much broader issue.But it should be noted that the data used are public and anyone can compute their own scores and that of others.It should also be remembered that the data are only for publications since 2001, which means highly cited papers before that were not included.This was done to standardise comparability and to measure research performance in recent years.This can result in some apparent anomalies in terms of the metrics for some eminent researchers, who may not appear so eminent because their most cited papers were published before 2001.
These results provide an objective and valid basis for comparing university performance from 2001 to 2013.There is much that can be done to extend this research and we have collected additional data for other countries.We are also in the process of gathering data on the prestige of the journal in which papers were published in order to make comparisons with the results of the citation analysis reported here.In addition, we are developing citation metrics that take into account both the quantity and quality of citations, by using both the direct and indirect citations to a paper, using the Google PageRank algorithm, as has been used to analyse publications in Physics (Chen et al., 2007).

Fig. 1 .
Fig. 1.The distribution of the hg-index across the marketing academics.

Table 1
Google Scholar benchmarks by academic level (hg-index) -top 500 universities in USA, Canada, UK, New Zealand and Australia.

Table 2 Average
department hg-index for full professors by country (departments with ≥ 4 full professors).

Table 3
Departmental net impact scores by country.

Table 5
Department mean hg-index scores by academic rank for ANZ universities with 4 or more marketing academics.