Reality-check : Cost-related journal assessment from a practical point of view 1 , 2

Journal acquisition management is no bed of roses. In times of global economic crisis, stretched budgets and price inflation, library managers try to square the circle by satisfying the continuously increasing demand with steadily decreasing resources. The phenomenon that prices for scholarly journal subscriptions rise much faster than the rate of inflation has coined the term ‘serials crisis,’ a problem that goes back to the 1980s (Houbeck, 1987). Many libraries all over the world cannot keep up with these price increases and the permanent proliferation of scientific output. The logical response has always been the cancelation of nonor less-needed subscriptions, with the effect of new price increases at the publishers’ end to compensate their loss–a picture-perfect example for a vicious cycle. Librarians too try to shift their budget in order to maintain the scope and depth of their e-library, however, ‘it is hardly sustainable for libraries to rob all the possible Peters (the monographs budget, the equipment budget, the personnel budget, etc.) to pay Paul (the electronic serials budget) and thereby protect an ever-smaller core of resources’ (Feather, Bracken & Diaz, 2008). In order to relieve the distress of the librarian community, publishers introduced the big deals–huge bundles of journal titles as ‘one size fits all’ packages. In *Corresponding author. E-mail: christian.gumpenberger@univie.ac.at; juan.gorraiz@univie.ac.at, ambros.wernisch@univie.ac.at


BACKGROUND
Journal acquisition management is no bed of roses.In times of global economic crisis, stretched budgets and price inflation, library managers try to square the circle by satisfying the continuously increasing demand with steadily decreasing resources.The phenomenon that prices for scholarly journal subscriptions rise much faster than the rate of inflation has coined the term 'serials crisis,' a problem that goes back to the 1980s (Houbeck, 1987).
Many libraries all over the world cannot keep up with these price increases and the permanent proliferation of scientific output.The logical response has always been the cancelation of non-or less-needed subscriptions, with the effect of new price increases at the publishers' end to compensate their loss-a picture-perfect example for a vicious cycle.Librarians too try to shift their budget in order to maintain the scope and depth of their e-library, however, 'it is hardly sustainable for libraries to rob all the possible Peters (the monographs budget, the equipment budget, the personnel budget, etc.) to pay Paul (the electronic serials budget) and thereby protect an ever-smaller core of resources' (Feather, Bracken & Diaz, 2008).In order to relieve the distress of the librarian community, publishers introduced the big deals-huge bundles of journal titles as 'one size fits all' packages.In spite of the fact that these arrangements are beneficial in terms of extended access, they offer less cancellation flexibility and bear the risks of weakening collections with unwanted titles as well as becoming more dependent on monopolizing publishers (Frazier, 2001).Baker (2008) clearly criticizes big deals as not being sustainable in their present form.He mentions the Pareto principle of 80/20 to illustrate that a substantial percentage of purchased bundle content is actually never accessed and that 'there is a limit to how much even a fat man can devour from the all-you-can-eat model' (Baker, 2008).According to Derk Haank (CEO, Springer) the big deal has solved the serials crisis and can be regarded as 'the best invention since sliced bread' (Poynder, 2011).However, Prosser (2011) paints a somewhat gloomier picture in his article about the current situation in the UK, which is certainly also true beyond UK borders.He describes the dysfunctional nature of this very market, where customers can only respond weakly to price changes, because journal content is non-substitutable.Therefore, customers feel the pressure to subscribe to certain titles irrespective of price.'Unfortunately, the only way to get the cheapest costs per download is to take the big deals.The "per serving" price of these all-you-can-eat options may be affordable, but the total price certainly is not' (Prosser, 2011).
Independent of the applicable acquisition model-singletitle subscription, small bundle or big deal-library managers are forced to focus on economic aspects and aim to get the best value for money.They need to weigh up costs against relevancy with regards to content.
Bibliometric indicators offer possible guidance in assessing scholarly journals.Since the development of the classic (journal) impact factor by ISI in the 1960s, the world has seen a constantly growing number of methods and indicators, which on the one hand offer a bigger choice, but on the other hand complicate the whole evaluation process.A communication gap exists between bibliometric researchers and the intended (but overwhelmed) users of the developed methods and indicators.Therefore, the evaluation is still often restricted to the flawed impact factor, since this indicator is well established and easily accessible (Haustein, 2011).
The standing of a journal within the scientific community can certainly not be based on one single indicator that only relies on citations.A multidimensional approach has therefore been suggested, which includes 'journal output' (number and length of articles and issues, distribution of document types, number and age of references and internationality of contributing authors), 'journal content' (analysis and comparison of published topics, thematic specialties, emergence of new research areas), 'journal perception' (usage measured by downloads, click rates and social bookmarking), 'scientific communication' (traditional citation analysis) and 'journal management' (editorial policy, review process and pricing).Haustein (2011) applied this multifaceted approach by way of the example of 45 physics journals.Library managers mostly lack the time and expertise to perform in-depth multidimensional scientometric journal evaluations.That is why single-indicator based assessments in the decision-making process are still predominant.Therefore, price-related indicators that are easy-to-use in daily life and provide crucial information to determine the value of a journal are appreciated.The related Web sites www.eigenfactor.org(West et al., 2011) and www.journalprices.com(Bergstrom & McAfee, 2010) offer such price-related metrics with according definitions, calculations and search functionalities, which are openly and freely accessible.
Eigenfactor was developed by Carl Bergstrom (2007) based on the PageRank algorithm.It measures the importance of a journal within the network of academic citations and takes into account the prestige and type of the cited and citing documents as well as subject-specific citation patterns.'If one is interested in asking what the total value of a journal is-in other words, how often our model researcher is directed to any article within the journal by following citation chains-one would use the Eigenfactor score' (West, Bergstrom & Bergstrom, 2010b).The additive property of Eigenfactor score is especially useful for librarians in acquisition management dealing with the analysis of bundles, since the Eigenfactor score of a bundle is simply the sum of all Eigenfactor scores of its constituent journals.By dividing costs (subscription price) by Eigenfactor score one can determine the cost-effectiveness of a journal or according to the additive property also of whole bundles (West, Bergstrom & Bergstrom, 2010b).
Further, price-related and Eigenfactor-independent metrics like 'price per article', 'price per citation,' 'composite price index (CPI)' and 'relative price index (RPI)' are provided on journalprices.com(Bergstrom & McAfee, 2010).In order to calculate the latter, the previous three values need to be known.RPI is of most interest to librarians, since its 'value' category allows broadly categorizing a journal as 'high value,' 'medium value' or 'low value' journal.
Research value based on Eigenfactor score and pricing value based on the before mentioned Bergstrom/McAfee categorization were already successfully applied to a journal assessment of the chemistry discipline performed by the Ohio State University Libraries which also considered faculty feedback and usage data (Feather, Bracken & Diaz, 2008).

PURPOSE OF THE STUDY
The University of Vienna is the oldest university in the German-speaking world and one of the largest in Central Europe.With more than 90,000 enrolled students and a staff of more than 9000 employees (6700 of these are academic), the Vienna University Library faces the challenges to provide all the needed information resources in the most cost-effective way in times of global economic crisis.There is only restricted flexibility in regards to publisher agreements (especially big deals).Therefore, informed decisions are crucial to use the little scope left in acquisition management in their very own interest.
In spite of the fact that librarians were the first to apply bibliometrics in practice (Gross & Gross, 1927), most of them are normally no expert scientometricians.For this reason, journal assessment procedures are supposed to be straightforward.Acquisition managers need to know if they provide the right content for the best money.The corresponding indicators should be easy to determine and support the decision making in what to keep and what to cancel.
Our study explores a possible approach for journal evaluation suitable for librarians.The focus is on practical orientation, and the Vienna University Library serves as an example.
On the one hand the assessment aims to disclose the relationship between several defined journal subsets regarding cost, usage, citation rate and publication preference.On the other hand cost-related metrics provided at eigenfactor.organd journalprices.com are tested in matters of reliability and usability as valuable tools for acquisition management purposes of an academic library.

Determination of the Most Expensive Journals
In order to determine the 'most expensive' journals one has the 'agony of choice' due to many different journal acquisition models and the alternative 'pay per view' option (which is also very inconsistent).Unfortunately there is no perfect, generally valid method to determine the effective cost of one journal.As a pragmatic approach the following two methods were applied:

Method 1
The prices are based on the information gained from journalprices.com (as of July 2011).
As a sample, the 100 most expensive titles in the category 'price per article' that are available at the Vienna University Library in any form (print or online, regardless of acquisition model) were selected.The limitation of this method is the strict validity for single-title subscriptions at list price only.

Method 2
The prices are based on the 2011 local subscription data of the Vienna University Library.
As a sample, the 100 most expensive single-title subscriptions were selected.The limitation of this method is the fact that many pricy titles (some of them of high impact) are purchased in bundles.

Determination of Journal Titles with the Most Downloads
A sample of the 500 most downloaded journal titles at the University of Vienna in 2010 was put together based on publishers' COUNTER reports (Project COUNTER, 2011) provided by the eResource Management Department of the Vienna University Library.

Determination of the Most Popular Journals to Publish in at the University of Vienna
Since there is no database available that completely covers all disciplines, Web of Science (WoS) and Scopus seemed to be the most appropriate choices to identify the 'most published in' titles at the University of Vienna.
Affiliation-based expert queries were performed in Web of Science (WoS) and Scopus for the publication years 2006-2010 in order to retrieve the top 100 journal titles in each database.
The results of both queries were compared.Due to the good match of results obtained from both databases, WoS was chosen as a pragmatic solution for the remainder of the study.

Determination of the Most Cited Journals in Vienna University Publications
The most cited journals in Vienna University publications of 2010 were determined by retrieval of the corresponding pub lications in Web of Science (analogous search method as before) and a cited reference data analysis by means of Bibexcel.

Overlap of Subsets, Calculation of Correlations
Each journal in all 3 subsets was assigned a rank for each of the evaluated criteria.Co-occurrences between the top ranks were identified.All correlations were calculated using Pearson's coefficient for the data rows as stated in each case.The results were rounded to two decimal places.
Comparisons were made for the following pairs of subsets: • most expensive vs. most downloaded

Determination of Cost-effectiveness
Cost-effectiveness (defined as price divided by its Eigenfactor score) was calculated exemplarily for the Vienna University specific AIP (American Institute of Physics) and NPG (Nature Publishing Group) bundles.Results based on price information retrieved from the publishers' Web sites (list prices) were compared to those based on the prices actually paid at the Vienna University Library 2012.
In order to obtain the aggregate Eigenfactor score for a bundle the Eigenfactor scores of all constituent journals were summed up.
List prices (originally either in USD or GBP) were converted into EUR (exchange rates as of 17 February 2012, Oesterreichische Nationalbank).
Due to confidentiality reasons, effective Vienna University Library prices cannot be disclosed.

Determination of Relative Price Index (RPI)
The relative price index (RPI) is obtained by dividing composite price index (CPI) by the average CPI in its subject category.The CPI itself is defined as the geometric mean of the 'price per article' and the 'price per citation.'RPI < 1.25 is considered as 'good value,' RPI > 2 as 'bad value,' everything in between as 'medium value.' Unfortunately neither a description on how to calculate 'average CPIs' nor 'ready to go' CPI values are available at journalprices.com.
The aggregate RPIs respective value categories were calculated exemplified by using the Vienna University specific AIP (American Institute of Physics) and NPG (Nature Publishing Group) bundles based on journalprices.com2010 data.As Bergstrom & McAfee (2010) do not propose a specific formula for a bundled RPI, three different computation methods were tested and compared.

Method 1
The aggregate RPI is determined by the average (arithmetic mean) of all RPIs of the single titles in a bundle.The values are categorized as explained before.

Method 2
The aggregate RPI is obtained by the geometric mean of all the single titles' RPIs in the bundle.The values are categorized in the same way as for method 1.

Method 3
Each journal (with valid data from journalprices.com) in the bundle is assigned a number of points depending on its respective value category: 3 points for 'good,' 2 points for 'medium,' and 1 point for 'bad.'These points are summed up and divided by the number of the analyzed journals in the bundle.A bundle is categorized as 'bad' for a ratio less or equal 1.67, 'medium' for a ratio higher than 1.67 and less or equal 2.33, and 'good' for a ratio higher than 2.33.
In addition to the bundle calculations, the top 100 'most expensive' single-title subscriptions were analysed according to their RPI-based value category.

FINDINGS 'Most Expensive' Journals
According to method 1 (price per article), the top 10 high-cost journals belong to the subjects Engineering, Physics, Biology, Business and Economics ( categorization from journalprices.com); seven titles are published by Emerald, two by Elsevier and one by Taylor & Francis.
According to method 2 (single-title subscription price for Vienna University Library) and compared to the results from method 1, the top 10 high-cost journals additionally belong to Chemistry and Medicine, whereas none of these titles is assigned to Biology or Economics.The picture is different regarding the publishers as well: eight titles are published by Elsevier, one by Emerald and one by Taylor & Francis.Both top 10 high-cost journal lists are completely different from each other regarding their composition of titles.
The intersection set of both samples (100 most expensive journals according to 'price per article' and 100 most expensive single-title subscriptions) contains only two titles: 'Critical Reviews in Environmental Science and Technology' (ranking position 29 according to 'price per article' and 70 according to subscription price) and 'Advances in Physics' (ranking position 3 according to 'price per article' and 92 according to subscription price).
Nature, Science and PNAS have ranking positions >200 and are not among the analyzed sample of most expensive 100 journals according to 'price per article'.

'Most Downloaded' Journals
According to the information gained from the different COUNTER reports the top 10 'most downloaded' titles all either belong to Physics, Chemistry or Biology (categorization from journalprices.com).Nature, Science and PNAS are the top 3 'most-downloaded' titles.Five out of the top ten titles are society journals.

'Most Published in' Journals
The congruence of the top 10 'most published in' title lists obtained from WoS (2006WoS ( -2010) )  The four journal subsets have not one single title in common.Excluding the 'most cited in Vienna University publications' subset, the number of shared titles for the remaining three subsets is still only three.

Correlations
The results of all compared subsets are summarized in Table 1.

Cost-effectiveness
The different results for the cost-effectiveness analyses of the Vienna University specific AIP and NPG bundles are comparatively shown in Table 2.

Relative Price Index (RPI) and Value Category
The RPI values calculated by using the three different approaches are comparatively shown in Table 3.
The RPI value categorization for the top 100 most expensive single-title subscriptions of the Vienna University Library is as follows: 13 'good value,' 20 'medium value,' and 60 'bad value.'

Determination of the Most Expensive Journals
Acquisition managers face an abundance of challenges when trying to identify the most expensive journals in their collections.It is a question far from trivial to answer, since there is no consistent pricing model that can be easily applied in general.Certainly list prices for single journals or packages (bundles, big deals) exist; however, their relevance is limited.Institutional prices are mostly different and depend on many factors like 'electronic vs. print,' 'individual institution vs. consortium,' 'online license with or without archival rights' and 'tiered pricing according to institutional size (FTE)' to name but a few.
The two described approaches in this study (1) according to journalprices.com and (2) according to effective prices for single-title subscription at the University of Vienna represent only a selection of the broad spectrum and obviously produce different results.The first approach has the advantage of independence from the acquisition model (therefore also includes titles acquired in bundles), but has the disadvantage of not referring to effective institutional prices.Whereas the second approach exactly does so, but excludes all costly titles, which are acquired in bundles.Unfortunately there is no ideal single approach, and specific methodological limitations always need to be taken into account when interpreting the obtained results.

Determination of the Most Downloaded Journals at the University of Vienna
The need for appropriate usage metrics is evident from several current initiatives (Gorraiz & Gumpenberger, 2010;Bollen & van de Sompel, 2008;Shepherd, 2007).This study relies on COUNTER statistics, which certainly come with several limitations (Lorenz, 2010;Baker & Read, 2008).First of all there is the dependency on the publishers and therefore no control over usage tracking.Second, COUNTER statistics reflect access per title in the current calendar year and provide no information about the accessed publication years.Unfortunately it is impossible to distinguish the usage of old and recently published content.Furthermore inconsistent assignment of usage data occurs in cases of title changes.Nevertheless the COUNTER reports are still the prevailing data sources for tracking the usage of electronic journals and therefore highly influence any imminent cancelation decisions.

Correlations
According to the obtained results, the relation between journal price, journal usage and journal publication preference is negligible to non-existent (taking into account the methodological shortcomings mentioned for price determination and usage tracking).Introducing citation rate as an additional dimension, a very weak correlation has been found between total cites and publication preference and a weak correlation between Viennese citations and publication preference.The latter is interesting as it reflects to what extent an institution returns gained and assimilated knowledge to the same journals in the form of citations, what we called 'Journal Prestige Transfer Rate' (JPTR).
A high correlation expresses a high success rate of an institution to not only cite but also publish in these cited prestigious journals.This could be a valuable indicator to measure the degree of knowledge transfer and is currently being investigated further in follow-up studies.
Finally, the correlation between citation rate and journal price turned out to be weak as well.
A really significant correlation is only evident between citation rate and journal usage.The calculated correlations between two kinds of citation rates (total cites as well as citations by the University of Vienna) and journal usage were almost identical.This again suggests a high correlation between 'Total Cites' and 'Eigenfactor score' itself (Davis, 2008;critical: West, Bergstrom & Bergstrom, 2010a) and is valuable information for all acquisition managers who have no access to the Thomson Reuters product and need to restrict to freely accessible Eigenfactor scores via eigenfactor.org.

Practicality of Eigenfactor-based Cost-effectiveness and Non-Eigenfactor-based Relative Price Index (RPI) for Acquisition Management Purposes of an Academic Library
Both eigenfactor.organd journalprices.comoffer costrelated metrics for journal assessment.These metrics are promising at first glance, but turn out to be far from feasible when applied to real life assessments like in this study.In general the information provided on both Web sites is usable for single titles and given prices.Issues arise as soon as one tries to calculate with effective prices, especially when going beyond the single title entity and applying the metrics to bundles.
First of all there are discrepancies between both related websites.The most recent data for cost-effectiveness calculations on eigenfactor.orgare from 2009, whereas journalprices.comrelies on data from 2010.
Cost-effectiveness on eigenfactor.orgonly offers a search interface with no further explanations or definitions of calculation (only to be found at journalprices.com).Furthermore the search interface does not allow searching single journal titles.They can only be identified after doing either a category or publisher (entries not normalized) search and browsing through the list of results, which is quite cumbersome.
According to an email from Jevin West (contact for eigenfactor.org)the metric for cost-effectiveness was recently altered to now using normalized prices.This new calculation would of course be much easier for interpretation and comparisons.However, its description (formula) is not to be found anywhere in both Web sites.Lacking this information the cost-effectiveness values obtained are always absolute values, which are difficult to interpret without any reference value.In this study the 'old' cost-effectiveness (price divided by its Eigenfactor score) was calculated either based on list price or effective price.This way it was at least possible to compare both values for the analyzed bundles.
The situation is similarly unsatisfying for the calculation of the relative price index (RPI), which would also be a very useful indicator apart from the Eigenfactor-based cost-effectiveness.For a librarian in acquisition management, it is more than interesting to know whether a journal offers good, medium or low value.Journalprices.com again provides 'ready to go' RPI values for single titles, it also provides the formula for the calculation.However, in order to determine the RPI independently one would need to know the 'average CPI of non-profit journals in the corresponding subject category.'Unfortunately no such information is provided nor are the average CPIs for the different subject categories per year given.
Nevertheless it was tried to determine the value categories for both analyzed bundles in the study by using three different approaches, which more or less produce the same results.The rating system of method 3 (see Methods and Procedures) was inspired by the journal assessment conducted by the Ohio State University Libraries (Feather, Bracken & Diaz, 2008).The limitation of all 3 approaches is the dependence on given RPI values based on pricing information from 2010, thus a different picture would probably be obtained when using RPIs based on effective prices.

CONCLUSIONS
This study aimed to assess the relations between journal cost, journal usage, publication preference and citation rate.Correlations could only be found between citation and download rates, which confirms that institutions actually download the publications that they are truly citing.
The low correlation between journal usage and journal publication preference has already been reported in other previous studies.Gorraiz & Wieland (2009) challenge financing models like SCOAP3, which aim to distribute the publication costs in accordance with the number of documents published by authors, institutions or countries.The correlation between citations and publication preference is more adequate to be used in order to measure the performance of an institution and is suggested as a possible new indicator, the JPTR, 'Journal Prestige Transfer Rate.' Costrelated metrics from eigenfactor.org( cost-effectiveness) and journalprices.com(value according to relative price index) were tested regarding appropriateness and usability for journal evaluations in acquisition management.The timeliness for this desire could not be better in the light of the current worldwide boycott against Elsevier, which is certainly only the tip of the iceberg: the gap is widening between the publishers' financial interests to maximize their profit and the utopian if not naïve aspirations of the scientific community to exact fair pricing.Considering inconsistent or even outdated available pricing information on the one hand and the complexity to easily determine effective institutional prices on the other hand, the use of the tested metrics is unfortunately very limited in practice.Furthermore explanations for calculations are incomplete and entirely exclude the prevailing acquisition practices of bundled content.Multidimensional journal evaluation is certainly desirable in theory, but hardly feasible in practical acquisition management.
• most published in vs. most downloaded • most published in vs. most expensive • most downloaded vs. most cited in Vienna University publications • most expensive vs. most cited in Vienna University publications • most published in vs. most cited in Vienna University publications