Assessment and Evaluation

Scarcely a decade into the digital library environment, librarians already know considerably more about digital library use than they did about traditional library use in the print environment.  In traditional print-based libraries, librarians counted outputs such as circulating library materials, reference and information questions, and interlibrary loans to and from other libraries.  In retrospect, the data collected was not reliable and, most likely, inconsistent due to varying loan periods, local practices regarding how to count informational and directional versus reference questions, and variances in how libraries classified interlibrary loans as opposed to circulation transactions.


Prior to the advent of online catalogs and integrated library systems, even circulation data for specific volumes or classes of materials was difficult to compile and analyze. Books were added to libraries’ collections, but detailed information about their subsequent circulation patterns was not easily available until libraries began to automate the library equivalent of electronic inventory control systems in the last quarter of the twentieth century.  The data collected by the automation of cataloging and circulation systems made it easier to count collection size and circulation, and break them out by subject.   Journal review projects in the print environment were predominantly in response to budget crises, were undertaken largely to cancel titles that were perceived as less frequently used and/or seemed overpriced and were frequently subject to manipulation by either librarians/selectors or the users they consulted.


Before the emergence of digital libraries during the last decade, librarians evaluated collection usage data, when they were: (a) interested in measuring their libraries’ performance, (b) asked to compile statistics for professional associations or governmental agencies, or (c) when confronted with budget cuts, librarians had to determine how the collection was being used.  Librarians typically relied on gross circulation counts and routinely employed unscientific and unreliable sampling plans and simple in-house data collection methods such as asking users not to re-shelve library materials so the library could count them.  These usage studies purported to measure library collections use when in fact there was never any tangible proof or consistent interpretation of what a book being removed from the shelf, or even a circulating item, really represented.


Looking back, collection development in the print environment was more of an art than a science.  Libraries knew how much they were spending, but were unable to ascertain how their collections were being used or how to use the data they could collect to better inform purchasing decisions.


It is telling that the authors of one of the most commonly cited articles on print collection use in an academic library, published in 1977, “Use of a University Library Collection” observed that:

…the gross data available up to this point have been too global in character and too imprecise in nature to serve as an adequate basis for the reformulation of acquisitions policies.  It is not particularly helpful for a bibliographer to know that ten percent of the titles selected will satisfy 90 percent of client demand for materials in a given discipline, unless we can determine which ten percent.  It is useless to tell the acquisitions librarian that half the monographs ordered will never be used, unless we can specify which 50 percent to avoid buying.  (Galvin and Kent, 1977)


Automated library systems changed librarians’ ability to review acquisitions decisions, at least for monographs. In 2003, a Mellon Foundation-funded study by the Tri-College Library Consortium (Bryn Mawr, Haverford, and Swarthmore Colleges) done in conjunction with the Council on Library and Information Resources found that approximately 75 percent of the items in the three libraries’ collections had circulated one or fewer times in the past ten years.  Also, about 40 percent of the items in the collections overlapped (i.e., they were held on more than one campus).  About half of these overlapping items had not circulated in the past 11 years.


Galvin and Kent referred to the book budget in the academic world as “the most sacred of sacred cows” and pointed out:

The hard facts are that research libraries invest very substantial funds to purchase books and journals that are rarely, or never, called for as well as equally large sums to construct and maintain buildings designed to make accessible quickly titles that are no longer either useful to or sought by their clientele.[i]


Galvin and Kent’s and the Tri-College Library Consortium findings were undoubtedly distressing to librarians, who typically based their selections in print-dominant libraries based on their experience and training to correlate the literature of various genres and disciplines with their users needs and interests in those fields.  Librarians were considered the “experts” at building collections and their purchasing decisions went largely untested and unquestioned. At the same time, librarians and their sponsoring organizations prided themselves on the size of their print collections, adding shelving at the expense of user spaces and building additions, new libraries, and high density storage facilities to house their print library collections.


The older print focused model looked at the collection from the point of view of the collection as a resource.   Now as the new ARL statistics show, the collection is increasingly a service, and data collection is guided toward user outcomes.



Networked electronic resources and services for library patrons have become more pervasive, easier to use, and have increased in number and variety since the days when patrons used predominantly print resources.   As libraries devote increasingly large proportions of their materials budgets to networked electronic resources, the Association of Research Libraries recognized the changed landscape and launched its New Measures Initiative to measure and evaluate usage of networked electronic resources.

According to ARL Statistics, 2005-2006, expenditures for electronic resources among the major research libraries in North American was more than $400 million, and after including hardware and software and other operational costs, the figure increases to half a billion.  (Kyrillidou and Young, 2008, 22-23).  This total represents close to 45% of the library materials budget.  The need to evaluate the return on the investment made on electronic resources was pressing.  Through the ARL E-Metrics new measures initiative (Association of Research Libraries, 2002), efforts were made to define the variables that would be useful to track and currently an experimental data collection is underway in the form of the ARL Supplementary Statistics that collected data on searches, sessions and downloads.

Article By :