Posted by: plinius on: oktober 14, 2006
Why is quality control so hard? Reference studies and reference quality in public libraries : the case of Norway. Paper for IFLA Library Theory and Research Section, Berlin 2003.
Summary
The quality of reference work in public libraries is highly variable. This is well known from the United States. Norway, Sweden and Denmark face the same problem. A Norwegian research project documented substantial flaws in the average level of service almost 10 years ago. But the results have not led to substantial corrective action. This paper asks why?
Our main conclusion is institutional. The library system lacks social mechanisms for learning from research. Today, the forces that promote change are too weak to transform established ways of working. Librarians know the evidence, but they do not act on it. Evidence-based librarianship requires a change in the relationship between research and action – which means a change within the professional identity itself. Serious attention to evidence is more likely to result from technology than from deliberate choice.
Public libraries provide a variety of services. One of them is reference. Libraries invite the general public to ask any questions they want – and take pride in providing professional answers on any topic. Reference work is not the most important library service, but it ranks near the top. In most library schools, training in reference subjects belongs to the core curriculum. A librarian without reference skills is an incomplete librarian.
We know, at the same time, that the quality of reference work is highly variable. In the United States, systematic studies of reference quality go back to the nineteen sixties. In Scandinavia, such studies only started in the middle eighties. But the results were similar: the chance of getting a full and complete answer was below 60% – and sometimes far below. In the first Norwegian study, which was undertaken in 1993 – see Salvesen (1994) – less than 30% of the test questions received adequate answers.
The Norwegian study was inspired by a Danish project from 1984. The Norwegian results were widely reported and discussed in Sweden – and led to a Swedish investigation using similar methods – see Jansson (1996). In all the Scandinavian cases, the quality levels fell below the US average. And this average – the famous 55%-rule – is itself dismally low. Effective libraries ought to give full and satisfying answers to 80-90% of all reference questions.
It should be mentioned, however, that nearly all studies of reference quality focus on a particular type of inquiry: factual questions with definite answers. In addition, survey questions tend to be more difficult than the average. Most factual questions in public libraries are easy to answer:
The Scandinavian studies used more difficult queries. This means that the 55%-rule has a bias. It does not apply to reference in general, but only to a subset of the queries. As Richardson (2002b) points out, reference is better than we thought.
|
UK |
Factual questions | Topical questions | Document questions | SUM |
| 1. Small public library – rural | 51% | 30 | 19 | 100 |
| 2. Small public library – urban | 43 % | 38 | 19 | 100 |
| 3. Large public library – urban | 38 % | 43 | 19 | 100 |
| 4. Very large public library |
28 % | 31 | 41 | 100 |
|
Source: Derived from Table 6.1.2. Percentage analysis of enquiry transactions by type, on p. 118 in England, Len and Sumsion, John. Perspectives of public library use. A compendium of survey information. Loughborough: Loughborough university, 1995. Libraries: 1 : Wincanton; 2: Queen`s Park; 3: Yeowil; 4: Westminster Reference |
||||
If we exclude directional and administrative questions (“where are the newspapers?”, “why must I have a card to borrow books?”, the three main types of reference transactions are
The relative proportions vary quite a bit between libraries. In a study of British public libraries, reported in England (1995), the percentage of factual questions decreased when the size of the library increased, from 51% in a small rural unit to 28% in the Westminster Reference Library in London (Table 1). Höglund (1997) found much lower rates for factual inquiries – below 2% – in a study of six public libraries in Sweden in 1995.
At Oslo Public Library a sample of one hundred physical questions in 1997 were distributed as follows (Høivik, 2000) :
topical – 56%; document – 27%, factual – 17%
These questions were addressed to a specialized reference section in Norway`s biggest public library. A study of five hundred digital questions in 2000-2002 gave the same ranking (Høivik, 2003):
topical – 71%; document – 19%; factual – 11%.
Difficult topical and document questions probably have higher success rates than difficult factual questions. This means that libraries have been tested under more stringent conditions than the average.
But in our context, this is not essential. The Scandinavian tests were not extreme. Professional service providers should be able to maintain their standards under moderate pressure. We must still conclude that public libraries did not achieve reasonable quality levels.
For the library communities in Scandinavia, the results came as a shock. The research was widely reported and hotly debated at library conferences. It was distributed through professional journals and report series. But what did actually happen in the service?
Basically : very little. Established forms of reference work continue. The information from the survey was clear enough. But nothing substantial happens. Practicing librarians do not deny the evidence. But they do not act on it. Library students learn about quality theory and quality management as part of their studies. But the knowledge is not applied.
After the project, some concrete steps were taken. In Norway, the original report was followed by additional studies. One of the project partners, the public library in Tønsberg, wrote a manual for improving reference services. They also introduced systematic quality procedures. For several years, there was a strong demand for training courses in reference interviewing.
But the total response, from the public library community as a whole, has not been strong. Only a handful of libraries have introduced some forms of quality control routines. At the national level, performance measurement was not a visible concern. On the whole reference continues as before.
I cannot speak for Denmark and Sweden, but I suspect their situation is similar. The quality evidence is neglected rather than rejected. In other words: research may show the need for change, but it does not in itself create change. The quality problem in public libraries is no longer a lack of insight, but a lack of action. The professional values are clear: the users deserve good answers. The empirical results are highly convincing: the users do not get what they deserve. Additional quality studies will hardly help. There is a gap between understanding and practice that must be bridged.
But why is it so difficult to act on the results? What makes effective quality control of reference work so hard to achieve?
The answer, I believe, ultimately lies in the nature of library organizations. Large organizations are not – in general – eager to innovate or learn. But organizational culture also reflects the social environment. In rapidly changing areas – the arts, media, front-edge technology – organizations tend to welcome innovation. In more stable milieus – insurance, railways, libraries – regularity and tradition reigns. Librarians love to speak about their roots: Alexandria, Athens and medieval monks. The past is always present.
Organizations in stable fields are inherently conservative. A suitable term is inertia. Newton`s first law of motion describes the inertia of physical objects. But it works for most organizations as well:
An organization at rest remains at rest. An organization in motion continues to move in the same direction – unless it is pushed off-course by a new force.
Most people and most organizations long for constancy rather than for change. We want predictable environments at work. We want mastery and control. Innovation is resisted rather than embraced. We want stability, which is another word for inertia. We want to be in charge.
Librarians are collectors and custodians of documents. And libraries are slowed down by the tremendous mass of their collections. The head of Oslo public library once described herself as the captain of a dreadnought. Orders may be given from the bridge, but a change of course, with one million volumes in tow, is a long and slow process. Inertia is a property of mass.
Large public libraries are hierarchical: a director at the top, department heads below, with ordinary librarians and library assistants at the bottom. Like hospitals and universities, libraries are professional bureaucracies. The power that comes from administrative superiority coexists with the power that comes from professional expertise.
Resistance to change is normal. Library rhetoric is full of “challenges” and “visions”. But the words are divorced from reality. Organizations in general, and library organizations in particular, seek constancy. They function like homeostatic systems: when they are pushed in a new direction, they push back.
Homeostasis needs no additional explanation. In organizations, inertia is the normal state of affairs. Reference work is no exception. The interesting question therefore becomes: how is institutional change at all possible?
The most obvious answer is technology. During the last thirty years, new information technology has transformed many aspects of library operations – including reference work. Yesterday, the speed and range of information retrieval was revolutionized by the new databases. Today, the web gives our customers direct access to information resources – and the demand for basic reference services is declining.
Technology-based change will continue. Tomorrow, print documents and physical loans will be the exception. Digital and virtual services will be the norm. But technology is a force that comes from the outside. It makes an impact whether we like it or not. Evidence-based librarianship represents something different: a deliberate approach to change.
If we look for agents or carriers of social change, we could – in principle – locate them in one of the following groups:
In public libraries, the customers have very little power. Firms that compete on the consumer market, must adapt to survive. An organization that lends books and provides answers for free, does not face the same exigency. Library clients do not organize themselves and demand better service quality. They do not sue the libraries for failing to answer their questions. Nor can they threaten to transfer their reference queries to a different provider. Public libraries are local monopolies.
In Norway, the government has started to take a strong interest in the quality and efficiency of municipal services. All municipalities are now asked to report detailed statistical service data, in a standardized form, to the Central Bureau of Statistics.
The Bureau has developed a sophisticated data base to store and present this information. Any person on the web can now use the data base to compare – for instance – the number of loans per staff year in different municipalities. Library statistics, which used to be of very limited political interest, can now be studied by local politicians and administrators. But for the time being I do not expect politicians to focus on quality in libraries. At the moment they have their hands full studying the quality of the school system.
When the political spotlight turns to libraries, the focus – I predict – will be on loans rather than reference work. It is easy to compile, to compare and to interpret lending statistics. The reference service is much more obscure. At the moment, our statistics on reference quantity (the number of reference transactions) are so poor that meaningful comparisons are impossible. Systematic data on reference quality are totally absent. The lack of routine reference data makes political intervention impossible. In five or ten years, this may change. But in the meantime, reference work can continue as before, quietly, in the background.
Quality is a key word in modern management thinking. The quality concept comes from industrial mass production. In highly developed countries, consumers are highly demanding. The mass markets are brutally competitive. The firms that produce cars, CDs or Coca-Cola for the masses must safeguard the quality of their products – in order to protect their market shares.
During the last 10-20 years, industrial quality thinking has increasingly been brought into the service sector – first into private, and later into public services. Library schools and library managers have taken the subject to their hearts. We are surrounded by tools and materials for quality management: courses, conferences, articles, books and guidelines.
Official documents tend to describe organizations in terms of a rational model. In rational organizations, change results from deliberate planning. The ideal change process follows a series of steps or phases:
In our case, this would mean the following:
But organizations involve interests as well as rationality. The implementation of new quality models often meets resistance from the staff. Since libraries are public institutions, employees have strong protection against dismissal. Since libraries are professional bureacracies, managers must share their power with the professions.
Professionals are not, in general, attracted to performance measurement. A profession is a group of persons that have a particular kind of training. The training is officially recognized and gives the candidates an exclusive right to certain positions or tasks. Professions are, in other words, small public monopolies. Within a profession, individuals tend to have substantial autonomy. Professionals are assumed to internalize the values of their profession. They do what is best for their clients because they are morally committed.
All quality models involve measurement. Quality is a property of the good or the service provided. In order to manage reference quality, we must know the quality level. Is it high or low? Does it go up or down? But once we start to observe quality, we are in fact monitoring the work of individual professionals very, very closely. Quality management threatens the cherished autonomy of professional people. Their office walls become transparent.
Professionals are not opposed to quality – but to measurement. They do not want to be monitored by administrators – like common employees. The evidence-based approach tries to avoid this dilemma – by locating the force of change within the profession itself.
Evidence-based librarianship is an intellectual import from another profession: medicine. Evidence-based medicine builds on two major ideas:
The first idea – research-based practice – goes back to the early 19th century. The second idea – systematic analysis – is quite recent (McMaster University, Canada, 1992).
If we say that librarianship should be evidence-based, we say that librarians should monitor, study and learn from the results of their own professional work. Evidence-based librarianship is a profession-oriented tool for change. Quality managers speak of learning organizations. Evidence-based librarianship suggests a learning profession.
EBM was an innovation. Evidence-based medicine represents a different way of doing medicine. Doctors that accept EBM, must change some of their customary practices. Doctors that work in teams, must change the practices of the team. In medicine EBM has been a success because it builds on established scientific methodology. Meta analysis as such is new, but the way of thinking is familiar to everybody with a research background.
In the library world, the problem is greater. Evidence-based librarianship would imply:
But in the library sector, the relationship between practice and research is weak. In most countries, librarianship is a craft or a semi-profession rather than a fully developed academic discipline. In Norway, the normal course of library studies takes three years. Our teaching is practical (or skill-oriented) rather than academic (or theory-oriented).
And this is a strength rather than a weakness. During the last few years, the library curriculum has become more theoretical and analytical. We now have a master program, and Oslo University College hopes to establish a doctorate program in the near future. But if most librarians had masters or doctorates, they would be poorly matched with the work most of them actually do.
Medical doctors must keep in touch with relevant medical research. They feel obliged – and are expected – to provide the best treatment possible. But medicine is a special profession. Since medical quality can be a matter of life or death, control mechanisms are exceptionally strong.
Libraries do not face the same pressures. Reference quality is not a matter of life and death. The consequences of low quality are less than dreadful. Librarians that fail, only waste the time of their customers. For the time being, reference quality is in the hands of the practitioners that populate the library system. Single studies, however shocking, are soon forgotten. Evidence is not enough.
From a professional point of view, the need for better reference services is evident. Libraries do not, as a whole, satisfy reasonable standards of reference quality. But the demand for improvement is low.
Public services improve when interest, standards, measurement and control develop together. But library authorities lack standards and statistics. Public libraries do not, in general, measure their reference activities. Local governments are not interested. They may care about their libraries, but they hardly care about the details of reference work as such.
The users of reference have limited leverage. As a group, they have an interest in reference quality. But since they only meet the library as individuals, there is no aggregation of their particular concerns into a collective demand.
Only the libraries and their managers have the power to act. And if they want to introduce formal quality procedures – including evidence-based methods – managers must cooperate closely with their staff. In professional bureaucracies top-down approaches are likely to fail.
In this paper we have looked at a particular case: reference quality research in Norwegian public libraries. In order to explain the lack of evidence-based action, we had to consider a more general topic, however: the forces that oppose – or support – change in organizations.
Here we face a thorny issue: the relationship between library research and library practices. Research is generally seen as an innovative force. We assume that research revealing weaknesses will be followed by policies promoting change. But this is not always the case.
In medicine and in professions like architecture and engineering the link between research and action is strong. The ways of working are standardized, and new scientific results are gradually translated into better procedures. Quality control is ingrained in the profession.
But in librarianship research and practice are only loosely related. I see evidence-based librarianship as an effort to create strongers link between research results and operational practice. But the task is not easy. The Reference Assessment Manual that was published by the American Library Association in 1995 is a good example
“In the past”, the authors note in the introduction, “evaluations and research in libraries tended to be conducted only in a single library with an instrument developed especially for the study and seldom utilized in other libraries. … Easy access and identification of appropriate evaluation instruments should encourage people to replicate and build upon previous studies”
The total research effort was, in other words, too scattered to make much of an impact. Five hundred projects do not establish a science. Scientific knowledge is not an aggregation of results. Scientific knowledge is a coherent body of concepts, methods, interpretations and results. Science is a collective undertaking. It proceeds by discussion, criticism and re-evaluation among researchers. Scientific interaction requires conflict as well as cooperation. Sometimes we must struggle with our peers. Sometimes we can build on their work. Library researchers must interact deeply in order to create a collective body of knowledge.
The final step is to use the evidence. Library science is an applied discipline. The purpose of quality studies is to improve quality. If the projects end with conference lectures, publications and degrees, they are incomplete. In an important new book, Richardson and Saxton (2002) have tried to review, and to draw practical conclusions from, all important studies of reference transactions in English. This is a big step forward.
In physics or anthropology, we cumulate knowledge. Knowing that something is just so has a value in itself. But library science is not a theoretical discipline. Librarianship advances when the practices – the state of the art – improve. Quality testing has no meaning unless it leads to quality improvement. In the applied sciences, we cumulate practices.
In the pure sciences, whether natural or social, the researcher struggles to understand the empirical world. In the applied sciences, whether natural or social, the researcher struggles to change the empirical world. In library science, this means improved practices. If library practices remain unchanged, why should researchers continue to polish their methods?
Traditional reference work is not very suitable for quality control. To be effective, standards require measurement. But typical reference transactions are – in general – too short, too private and too ordinary to be monitored.
The great majority of inquiries are answered in a few minutes. Clients are taken care of individually, on a drop-in basis. Reference appointments are rare. Practical reference work mostly takes the form of brief service encounters between two persons: the client and the librarian. There is hardly time for note-taking and written answers. Hence, there is no record of what goes on during the sessions.
It is not impossible to control quality under such conditions. Even brief encounters can be supervised. Burger shops and coffee bars do train their service staff in quick and friendly customer management. There is much libraries can learn about service levels from such extremely user-oriented operations.
But reference work is much less standardized. McDonalds`s may add chicken, fish and veggies to their portfolio – but the range of possible burger menus is still limited. Reference workers face a more complex situations. The variety of questions is endless. The number of relevant sources run into hundreds and thousands.
A recent Norwegian bibliography of current sources contains about 650 books – see Hald (1995). The Telephone Reference Service of New York Public Library draws on a Reference Room with about 2 000 volumes – not to speak of databases and the Web (Berliner (1992), introduction). Reference work, in other words, demands professional skills. Under such circumstances, defining, observing and imposing fixed quality standards is difficult.
Quality is not a free good: it comes with an effort. Perfection has a price. We are more willing to pay the price, if the lack of quality has serious consequences. Since we put our lives in their hands, doctors, nurses and airline pilots face strict quality controls.
Reference services are more humdrum and less consequential. The flow of questions reflect quite ordinary matters and concerns. We may be interested in the answers, but they are not essential to our well-being.
Digital technology changes the way we work. Librarians were early adopters of digital technology. Electronic data processing suited the kind of large scale, standardized data processing libraries needed. The MARC format was developed by the Library of Congress in the 1960s. Massive printed bibliographies were replaced by data bases. Vendors like University Microfilms International, Institute for Scientific Information and Dialog transformed the process of professional information retrieval.
In the 1980s, personal computers started to invade offices everywhere. The division of labor within offices started to change. The occupation called typist disappeared. Or rather: we became our own typists. People that use their own PCs for drafting letters and documents, why should they bring in another person at the very end?
Through the 80s and 90s library catalogues were gradually converted to data bases – or automated, as we used to say. This made lending operations much more efficient. But the big change in reference work comes from digital communications: with e-mail and the Web. Libraries that accept questions and provide answers by e-mail, have already taken the first step on the Moon. Digital transactions can easily be stored and retrieved, monitored and reused. The reference process is no longer inherently private.
The new technology has the potential to change the way reference work is organized. Questions and answers are normally recorded in digital form. This record – the reference log – makes supervision and control of quality feasible. The cost of data collection is low. This means that inquiries and responses can be sampled, studied and evaluated without time-consuming ad-hoc surveys.
The change from physical to virtual reference work creates new opportunities for quality control. Virtual reference facilitates a more team-oriented, dedicated and analytical approach to reference work. Librarians and library assistants can more easily share their knowledge. Supervisors can check what goes on day by day. Managers can analyze “inquiry trends” and react accordingly. Digital reference can actually be evidence-based,
But digital services will also reveal, I believe, current weaknesses in reference work much more clearly. At the moment, virtual inquiries constitute rather less than one percent of all reference transactions in Norwegian public libraries. The digital answers are, on the average, clearly better than the “physical” answers. But solid answers take more time. If library users were more aware of the quality gap, they would all “go virtual”. But that would sink the system.
In the United States, the Virtual Reference Desk Project (2000) has published a set of quality guidelines for its members. Guideline 11, on publicity, states:
Services should inform potential users of the value that can be gained from use of the service. A well-defined public relations plan can ensure that services are well-publicized and promoted on a regular basis. Publicity should not create more demand than the service has capacity to handle.
Librarians live in a strange world. If they improve their services, they should not advertize the fact. Will there never be quality for the masses?
The whole paper can be summarized as follows:
American Library Association (1995). The Reference Assessment Manual. Ann Arbor: The Pierian Press.
Berliner, Barbara (1992). The book of answers. The New York Public Library Telephone Reference Service`s most unusual and entertaining questions. New York: Simon and Schuster.
Hald, Odd Heide and Jofrid Holter (1995). Here is the answer! Handbooks and bibliographies for reference work [= Her er svaret! Håndbøker og bibliografier til bruk i referansearbeidet]. Oslo: Biblioteksentralen.
Höglund, Anna-Lena (1997). “Äntligen en riktig fråga! Undersökning av referensverksamheten vid sex huvudbibliotek i Östergötland. [Linköping]: Länsbibliotek Östergötland
Høivik, Tord (2000). Does the web have the answer? [= Har veven svaret? ]. Norsk tidsskrift for bibliotekforskning nr. 13, p. 9-25.
Høivik, Tord (2003). Why do you ask? Reference statistics for library planning. Performance measurement and metrics, vol. 4, no. 1, p. 28-37.
Jansson, Britta-Lena (1996). “This was difficult”. Reference quality in public libraries [= "Det här var svårt". Referenstjänstens kvalitet ved folkbiblioteken.] Lund: Rapport från Statens Kulturråd 1996:3
Richardson, John V. and Matthew Saxton (2002a). Understanding Reference Transactions: Turning an Art into a Science. Library and Information Science Series. New York: Academic Press.
Richardson, John V. (2002b). Reference is better than we thought. Library Journal 127, p. 41-42.
Salvesen, Gunhild and Synnøve Ulvik (1994). Does the library find the answer? Assessing reference service quality in Norwegian public libraries [= Finner biblioteket svaret? Utprøving av referansetjenestens kvalitet i norske folkebibliotek]. Tønsberg: Tønsberg bibliotek.
Virtual Reference Desk Project. Facets of Quality for Digital Reference Services. Version 4. October 2000 [URL = http://www.vrd.org/facets-10-00.shtml - Read June 7, 2003]
