Towards measuring the performance of public libraries in South Africa

Following the publication of the report of the Public and Community Libraries Inventory of South Africa (Paclisa), an issue for further investigation was raised given the difficulties that many libraries had experienced in completing the questionnaire. A Working Group on Public Library Statistics (WGPLS) was established to work on the drafting of a simpler data collection instrument. Our paper describes the process of arriving at joint understanding of the aims and objectives of measuring performance in the South African public library system. A brief discussion of the purpose and rationale of collecting statistical data, and their role as performance measures and indicators for public libraries in South Africa follows. The proposed statistical form and the indicators that could be derived are presented. The paper concludes with suggestions of future steps to be taken, including a consideration by libraries of outcomes, and the difference made by libraries to people’s lives.


Background to the investigation
Following the publication of the report of the Public and Community Libraries Inventory of South Africa (Paclisa), the Libraries Working Group of the Print Industries Cluster Council (PICC) raised an issue for further investigation.The Paclisa project had been set up with the following objectives: • Collect reliable data about each public and community library • Create a database about all the libraries • Map the libraries' location • Collect demographic data about the areas served by the libraries • Obtain market information on bookspend by libraries • Produce a directory of public and community libraries (Van Helden & Lor, 2002) The project, managed by the Libraries Working Group of the PICC and supported by a grant from the Carnegie Corporation of New York resulted in an inventory containing three databases: • A library directory containing identification and descriptive data about the libraries • A demographic database containing relevant demographic information • A geographic database containing geographical information such as municipal boundaries and location data for the libraries (Van Helden & Lor, 2002) From the data collection exercise it became apparent that many libraries experienced difficulties with completing the questionnaire.Many of the questionnaires were incomplete.The only questions for which high response rates were received were questions relating to number of books in stock, number of registered users and numbers of loans (Van Helden & Lor, 2002).This was on account of the complexity and length of the instrument, and the wide range of conditions and staff capacity in the respondent libraries.
The Carnegie Corporation of New York had agreed to make a grant to facilitate the drafting of a simple form for the regular collection of statistics from public libraries so that the database might be kept up to date.A Working Group on Public Library Statistics (WGPLS) was established by the Libraries Working Group, the team that had overseen the Paclisa project, to work on the drafting of a simpler data collection instrument.The intention of Paclisa was that the data should be collected at regular intervals so as to keep the database and directory up-to-date.The purpose of the WGPLS was to determine a set of generally agreed core public library statistics.The statistics should be of such scope that unqualified staff in small libraries would be able to collect them without requiring extensive training or intervention by the Head Office.The things or activities to be measured should be clearly and operationally defined so that uniform practice and comparability would be achieved.Guidelines and instructions for collection should accompany the core set of statistics, and terminology and definitions should be standardised.The main deliverables would be a statistical form and accompanying glossary and instruction manual to guide responding libraries in the methods of data collection to be used.

Performance measures
At this point it could be useful to step back for a moment and to distinguish between performance measures and indicators.Measures express aspects of a library service such as size, or amount of resources, or circulation which do not by themselves express much about performance.Indicators are derived from these measures by combinations, juxtapositions or ratios that express relationships between measures.Examples of such indicators could be the average number of books borrowed by each library user or the percentage of library members in a given population.
Performance indicators may primarily be derived from four types of measures: Input measures: such as staff, materials, funds Process measures: everything entailed in getting the material to the service point from which they are borrowed; the things we do to make materials accessible, such as the processes of acquisition, cataloguing and shelving Output measures: Products, services, activities, circulation, reference queries, interlibrary loans Outcomes: take-up of services, user satisfaction The brief of the Working Group was that public libraries in South Africa should all collect the same kinds of measures according to the same standards, eventually to enable libraries to derive indicators from these measures.The measures that were proposed for collection at this stage were input and output measures only; process measures, such as how long it takes to get a document delivered through ILL, or how quickly a returned item is reshelved, are generally regarded as for private and internal consumption, while outcome measures are more qualitative in nature and require techniques such as surveys, which may be done at individual libraries, as required.
Essential requirements for collecting standardized and comparable measures, are for standard definitions to be used in order to ensure uniform practice.Performance measures and indicators therefore have to be appropriate, reliable (unambiguous), reproducible, helpful and practical (Poll & Te Boekhorst 1996: 18-19).A glossary was also prepared to give definitions of measures and participants' opinions, comments and input were invited on the first draft.In its final form, the glossary additionally gave instructions for completing the form.
It is generally recognized that performance indicators enable libraries to address some or all of the following issues: • to manage effectively and to enable benchmarking • to improve their performance • to set priorities • to identify problem areas • to allow for comparability both across time and between institutions • to give a rounded picture of all aspects of library use • to enable local, state and national advocacy for investment in libraries to enable them to make an even better contribution to society and the economy To summarise therefore: "Statistical information is necessary to formulate sound policy advice, to analyse and predict trends, to monitor performance, to market effectively and to seek funding successfully" (Bundy, 2000).As competition for funds among service organisations has grown ever stronger in recent years, the need for statistical evidence of providing needed services, is "probably the most important reason for the increasing interest in effectiveness and performance measures" (Willemse, 1989).

Building consensus
At the end of 2001 an invitation was sent out to heads of respondent libraries to join the WGPLs or to send a nominee.Those who accepted the invitation, constituted, with the convenors, the Working Group on Pubic Library Statistics.Representatives from twelve provincial and metropolitan libraries joined the group.The WGPLS started work on 29 April 2002 with the launch of an e-mail discussion group, the initial vehicle for consultation and discussion among the members.It was agreed that a sensible starting point would be to analyze current practice and to see to what extent common ground was covered by all the libraries.Accordingly members submitted the statistical forms currently in use, and the convenors analyzed them with a view to isolating common elements to be incorporated in the first draft.The first draft of the statistical form was presented by the convenors at the first meeting of the WGPLS held during the Liasa Conference in Port Elizabeth on 3 October 2002.The draft form was compiled using the following methods: • Examination of the statistical forms sent by members of the WGPLS • Inputs and suggestions made by the members on the WGPLS e-mail discussion forum • Examination of the Paclisa questionnaire • Study of the international literature • Examination of various international standards • Examination of the National Center for Education Statistics annual report, Public libraries in the United States: fiscal year 1999 Using these sources, the convenors compiled a list of measurement categories, paying particular attention to the commonly used categories by participating libraries.This first draft statistical form was accompanied by a glossary.The workshop was useful in that participants discussed the purpose and rationale of collection of statistics, and their role in performance measures and indicators for public libraries in South Africa.Participants agreed on the following points of departure: • An important design principle would be the reasons why the data would be collected and the purpose to which the information would be put • It was recognized that there are different levels of operation (local and provincial) • There was an interest that we should aim eventually at international comparison • The best way of approaching the task would be to aim at identifying a core set of measures that everybody could agree on The convenors undertook to generate a list of measures and indicators which would form the basis for discussion at the follow-up workshop to be held towards the end of the year in Cape Town.The purpose of this workshop would be to consider a revised statistical form and glossary with a view to finalizing its content so that it could be distributed nationally.It was also agreed that clarity should be sought on the agency that would take responsibility for distribution of the forms and the collection exercise.
The second workshop was held in Cape Town in November 2002.It was agreed during discussions that a national information system for public libraries is necessary for policy making and for advocacy purposes.An essential requirement of the system, whatever its scope and degree of comprehensiveness, is that the data have to be reliable and comparable.The National Library of South Africa, and the Centre for Geoinformation Science at the University of Pretoria (the agency responsible for the design of the instrument and data collection in the Paclisa project) would assume responsibility for the data collection in 2003.
Some of the pitfalls of the use of data were aired.The view was expressed that some librarians are reluctant to collect statistics for fear that they might be used to the disadvantage of their library.This could lead to inflation of the figures returned.The concern about comparisons has been raised in the international literature, and is an issue that is related to the notion of national and international standards, a theme that surfaced in the workshop discussions.
For example, Thomas Hennen's annual publication of the "Great American Public Libraries: HAPLR Ratings" raises a great deal of interest among libraries and also in the media.Using data collected annually by the Federal-State Cooperative Service he ranks public libraries according to their score in a number of categories.His review is published in American libraries and always accompanied by vigorous debate, and variously defensive and triumphant responses.He notes in his latest review that German public libraries have produced a similar index, BIX, designed to produce comparisons of one library to another over time (Hennen, 2002: 68).
The United Kingdom introduced public library standards in January 2001, and libraries have three years in which to comply.It is intended that performance will be measured through a public audit process (Appendix 2: national standards).The standards cover: • Rules (e.g.book loan period) • Access (e.g.opening hours, geographic location) • Materials (e.g.minimum number of stock items added) • Quality of service (e.g. percentage of users who rate staff helpfulness positively) • Other aspects of service provision (e.g.number of qualified staff) • Outputs and outcomes (e.g.number of visits to libraries per 1000 capita) Participants at the workshop expressed the view that an unintended consequence of the potential for comparison among systems might be that a library whose spending is lower than another might be held up as efficient, rather than underresourced.Concerns were expressed about the bluntness of an instrument to sketch the use and value of a service.For example, many libraries have users and readers in greater numbers than registered borrowers.This raised the difficulty of counting, and the tension between efficiency and economy.It is far easier to count registered users than to count people using the library at any given time.Further evidence for this concern was reflected in a recent report of a meeting of the tenth annual general meeting of the friends of the Observatory Public Library in Cape Town.The chair of the Friends, Dr Helen Moffatt, voiced her opposition to the use of "the number of books issued (and cost per issue) as a means of calculating the cost effectiveness of libraries".She argued that this practice favours libraries that serve more affluent, literate and leisured communities, and observed that more than half of the regular visitors to the Observatory Library seldom take out a book (Passionate FOOLS keep the library doors open, 2003: 6).

Measuring performance of public libraries in South Africa
The original brief had been to draw up a form that was as straightforward to complete and as simple as possible.At the meeting in Port Elizabeth, participants were reminded that they would be collecting measures only and that the compilation of indicators was beyond the brief of the Working Group.A need was nevertheless expressed for a small number of indicators that would be more informative than simple measures, and it was agreed that a short list of possible indicators for optional use would additionally be drawn up for discussion at the next meeting, which was held in Cape Town in November 2002.
Participants were also reminded that is not possible to draw significant conclusions from measures and indicators collected over a short period, but over a number of years they could become important in showing the relative costs and the cost effectiveness of specific library services.It was therefore important to start collecting data immediately, and to continue doing so regularly.
The issue of whether or not to include measures reflecting costs elicited considerable discussion.The IFLA guidelines suggest that they should perhaps be left out.Some librarians may be afraid of including cost measures, as they may be used inappropriately, for example to demand the most service at the least cost (Te Boekhorst, 1995: 278).In addition, a particularly low-cost service may not be a very effective one or one that meets with the needs of the users (Poll & Te Boekhorst, 1996: 39).As was shown above, these concerns were indeed voiced by the participants.On the other hand, the print industry in South Africa was eager to have access to public library spending data, as were interest groups lobbying for issues such as the abolishing of VAT on books, or increasing spending on public libraries.Participants at both Workshops were therefore requested to consider whether or not to include cost data in the form.

The statistical form
The agreed statistical form was redrafted to take into account the participants' suggestions made in November.The final form was submitted for implementation to the National Library of South Africa in February 2003.The form is attached as Appendix A and the accompanying instructions for completion as Appendix B. As far as the question of including cost data was concerned, the participants agreed that it would be important to include a section on operating costs and one on expenditure (see sections 7 and 8 on the Form and the Completion Instructions).

Additional performance indicators that may be compiled from the proposed core measures
In response to specific requests from participants at the Port Elizabeth meeting, a document for the calculation of a number of fairly simple indicators, with brief descriptions of procedure was compiled and discussed.It was recognized that individual libraries might wish to calculate such indicators for their own purposes.These indicators, based on ISO11620:1998(E), were:

General library services
Percentage Some of the generally recognized library performance indicators such as In-house use; User satisfaction; Availability; and Seat occupancy rate, require special investigations involving sampling, which cannot be calculated from the suggested core measures.Individual libraries may choose to conduct such investigations and it is suggested that methodologies as set out in ISO11620 be used to ensure a standard approach.Members of the Workshop at the LIASA conference in Port Elizabeth had expressed particular interest in gathering data about in-house use which, it has been suggested, may well exceed circulation by as much as 50% (Western Cape, 2001:13).For this reason, two further indicators, also involving sampling, were suggested and explained, based on ISO11620: • In-library use per capita, involving a fixed sampling period during which users are asked not to reshelve documents used in the library.The used documents are counted and in-house use is calculated according to the suggested formula.• Document use rate, which involves estimating the proportion of documents in use at any one time and expressing the documents that are in use as a percentage of documents owned by the library.It was also pointed out that indicators are not always appropriate for reporting activities.For example the extent of Use of Extension Services (such as Exhibitions, Adult basic education classes, Lectures and talks, Story hours, Children's programmes, Reading circles and Video/film shows) are best reported in terms of the basic measures that simply state how many activities of each kind where held, and how many people attended: A total of ……people attended ….. activities of a particular kind.

Conclusion
So where do we go from here?The National Library of South Africa has started to distribute the questionnaires to public libraries throughout the country.But it must be clear that this is only the beginning.These are only the most basic measures, upon which one might begin to build a culture of assessment.It is most important to realize that input and output measures such as these give only the bare outline of what is happening at particular institutions and clearly they are not enough, or even most appropriate in all circumstances.They cannot give all the details and they certainly do not give any reasons why things are the way they are.For such explanations, a much rounder, fuller picture is required: for example, our membership is low, but our in-house use is very high, for the following reasons and with the following figures to back this up.Individual initiatives will have to be designed to address individual situations and needs.We need to start thinking about outcomes, about the difference we make to people's lives, and how to demonstrate this objectively.But it is also essential, and that is why these early steps are so important, to start somewhere, to see where we are according to the baseline measures, and to begin to fill in the rich picture from there.

Please study carefully before filling in the Statistical Report
The numbers refer to the numbers of the sections in the Statistical Report.Each section gives specific instructions on how to complete that section of the Report.PLEASE NOTE that one form should be completed for each individual library in a system.A separate form should additionally be completed by the Head Office of the system if any library activities (e.g.acquisition, cataloguing, or reference services) take place there.
1 Name of library: Give here the name whereby the Library is commonly known.Library code: Give here the SAIS (SA Interlending Scheme) number if applicable.Street Address: Full address details are particularly important if the Library has recently changed address.4.5 Give totals of all types of other materials acquired by the Library in the current financial year.

Circulation
5.1 Circulation of documents per annum is the count of all borrowing transactions or loans of library materials, including renewals.It is often regarded as a measure of the use made of library materials.Give total circulation figures where breakdowns into languages are not available.5.2 Give circulation figures of other materials that are available in the Library.5.3 OPTIONAL.In-house use.Especially in public libraries, circulation is often not a true reflection of the total use of a library's materials.It is therefore suggested that some Libraries might wish to measure their inhouse use, in which case it should be done as follows: A period has to be fixed for sampling.(Choose either one month or four separate weeks during the year.)During this period ask the users not to reshelve documents used in the library.Count every document before reshelving.An effort should be made to reshelve used documents as quickly as possible.Also calculate the number of days in the year that the Library was open.The formula for calculating annual in-house use will therefore be: number of documents counted in sampling period x total number open days per year number of open days in sampling period 6 Our official definition of interlibrary loan is "a transaction in which library material, or a copy of the material (including materials sent by telefacsimile or other forms of electronic transmission) is made available by one library to another upon request.It includes both lending and borrowing.The libraries involved in interlibrary loan are not under the same administration or on the same campus (ANSI/NISO Z39.7-1995)." For the purposes of this survey we wish to distinguish between documents requested by other libraries (lending) and documents received from other libraries (borrowing); and also documents borrowed to and lent from other libraries within a particular system (sometimes known as 'interbranch loans').These two sets of numbers should be recorded separately.
7 Operating Cost explores the sources of funds expended during the previous completed financial year.Please distinguish if possible between funds from provincial and municipal governments.All other sources of funds (e.g.cost recovery schemes, funds collected by Friends or other organizations, rent paid for use of facilities, etc.) should be added together and reflected in the "Other" column.If detailed breakdowns are not available, please supply the total amount for the LIBRARY only in the final TOTAL column.

Annual Public Library Statistical Report: Core Measures For Financial Year : ____
Members are persons or organizations that are registered with the library in order to use its documents within or away from the library.Give the total only if adults and children are not registered separately.3Users.A user is a person who has visited the library or used the services of the library during the past year.The number of users with registered loans should be used as an estimate of the number of users in the target population.document is a piece of recorded information which can be treated as a unit in a documentation process regardless of its physical form and characteristics (from ISO 2789).Treat all separate pieces of print information on paper as documents."Other" refers to all non-South African foreign languages.Give total numbers of documents only if breakdowns into different languages are not available.4.2 Number of other materials in stock: Give totals of all types of materials available in the Library.4.3 Number of documents acquired in the current financial year: Reference materials are to be included in the non-fiction blocks Give total numbers of documents acquired only, if breakdowns into different languages are not available.4.4 Give the total number of periodical titles currently received by the Library.Include newspapers in the count.