Assessing Quality in Digital Reference Services:
A Research Prospectus
Proposed by
Dr. Charles R. McClure, Francis Eppes Professor, and
Director
Information Use Management and Policy Institute
School of Information Studies
Florida State University, Tallahassee, Florida 32306
cmcclure@lis.fsu.edu
http://slis-two.lis.fsu.edu/~cmcclure/
850-644-8109
and
Dr. R. David Lankes, Assistant Professor and Director
Information Institute of Syracuse
Syracuse University
Syracuse, NY 13244
rdlankes@ericir.syr.edu
http://www.askeric.org/~rdlankes
800-464-9107
with support from
OCLC Online Computer Library Center, Inc.
6565 Frantz Road
Dublin, OH 43017-3395
1-800-848-5878
DRAFT
January 12, 2001
Overview
This prospectus seeks participants to fund a study to develop methods
to assess the quality of digital reference services, test and refine
measures and quality standards to describe digital reference services,
and to produce a guidebook that describes how to collect and report data
for these measures and standards. The project would begin in March 2001 and be completed in
January 2002. Libraries and other
organizations would subscribe to participate in the research and
development effort at a contribution of either $5,000 or $10,000+ each.
We intend to receive total commitments between $100,000 to $150,000 to
complete the project.
Each participating organization will receive regular status reports
on the progress of the project and may participate in the field-testing
of the measures and quality standards. The study will be conducted by
the Information Use Management and Policy Institute at Florida State
University under the direction of Dr. Charles R. McClure with the
assistance of Dr. R. David Lankes and the Information Institute of
Syracuse at Syracuse University with support from OCLC. The primary product from the study will
be a practical guidebook that will assist organizations in evaluating
digital reference services and producing measures and standards for
better planning and evaluation of digital reference services in
libraries.
Importance of Assessment and Quality in Digital Reference Services
At the October 2000 Virtual Reference Desk (VRD) Conference in
Seattle, the growing digital reference community identified assessment
of quality as a top research priority. As patrons demand more services
online, and as reference librarians seek to better meet patrons'
information needs through the Internet, it has become essential to
determine common definitions of success and quality. Library
administrators need strong, grounded metrics and commonly understood
data to support digital reference services, assess the success of these
services, determine resource allocation to services, and determine a
means for constant improvement of digital reference within their
institutions.
For the purposes of this prospectus, digital reference is defined as
human-intermediated assistance offered to users through the Internet.
Today, libraries are offering human-intermediated reference over the
Internet at an increasing rate. Research by Joe Janes and his colleagues
(Janes, 2000) found that 45% of academic libraries and 12.8% of public
libraries offer some type of digital reference service. These services
are often ad hoc and experimental. Janes and McClure (1999) found that
for quick factual questions, librarians using only the Web answered a
sample of questions as well as did those using only print sources. Many
libraries conduct digital reference service in addition to existing
obligations with little sense of the scale of such work or its strategic
importance to the library.
Further, there are no consistently held quality criteria in the
library profession for reference over the Internet. While some models do
exist, such as the Virtual Reference Desk's Facets of Quality (Virtual
Reference Desk, 2000), these criteria tend to be broad, non-library
specific and lack practical assessment methods and benchmarks. This
study will create the assessment tools to institutionalize digital
reference by developing a range of assessment techniques, measures of
services, and quality standards.
Previous work by McClure (Bertot, McClure, and Ryan, 2000) suggests
that assessing the quality of services in an electronic or networked
environment is complex and requires multiple methods of assessment.
Further, for the assessment techniques to be useful in a library
setting, the procedures and methods need to be practical and easily
implemented. Recent work in measuring and improving customer
satisfaction suggests that there has only been limited attention to
quality services and standards in libraries (Hernon and Whitman, 2000).
Nonetheless, there is an evolving body of knowledge about designing and
implementing "quality"-based services. Bertot, McClure, and
Ryan (2000) and Hernon and Whitman (2000) provide a useful starting
point for developing measures and standards for digital reference
services.
This study will examine the following areas:
- Quality criteria and benchmarks for digital reference services as
well as the growing number of digital reference networks and
consortia,
- Economic models and costing metrics for digital reference services
and networks,
- The use and importance of the human intermediary in digital
reference services (LITA, 1999),
- Strategies for enabling librarians to assess electronic and
networked resources for current and new services and describing
digital reference work in economic and other terms.
Assessing Digital Reference Service
Quality assessment for digital reference can be divided into a number
of measurement components. For this study, however, we will concentrate
on the following:
- Outcome measures (quality of answers): accuracy of
responses, appropriateness to user audience, opportunities for
interactivity, instructiveness, and impacts resulting from the
digital reference process.
- Process measures (effectiveness and efficiency of process):
service accessibility, timeliness of response, clarity of service
procedures, service extensiveness (percentage of questions
answered), staff training and review, service review and evaluation,
privacy of user information, user awareness (publicity).
- Economic measures (costing and cost effectiveness of
digital reference): the cost to conduct a digital reference session,
infrastructure needed to support quality digital reference services,
and impact of these costs on other library expenditures.
- User satisfaction (degree to which users engaged in digital
reference services are satisfied with the process and the results):
satisfaction indicators can include accuracy, timeliness, behavior
of the staff, technical considerations, physical facilities, and
others.
These categories can overlap since measures can describe multiple
components. While the study will emphasize measures and quality
standards in these areas, the specific measures and quality standards
that are ultimately proposed will depend on field-testing and the
investigators' abilities to proceduralize the method to produce the
measures and standards in a practical and reliable manner.
As a point of clarification, we use the term statistic as a
descriptive count describing an activity or outcome (e.g., number of
e-mail-based reference questions asked). A measure relates descriptive
data in such a way that additional insight can be obtained about the
activity or outcome (e.g., correct answer fill-rate, or the percentage
of all reference questions asked that were answered correctly). A
quality standard is a value statement of the level or quality of
services that should be provided (e.g., librarians at the Edward H.
Jones Memorial Library will provide an 85% correct answer fill rate
within 24 hours of the time the question is asked). Again, terminology
is oftentimes not consistent in this area and distinctions among
statistics, measures, and quality standards can be blurred (Kasowitz,
Bennett, and Lankes, 2000).
Study Goals and Objectives
The overall goal of the project is to better understand and describe
the nature of quality digital reference services. Specific objectives
include:
- Develop a model that helps to describe and explain the basic
components that comprise digital reference services by reviewing
existing sources of information (including best practices at
participating organizations).
- Propose measures and quality standards based on this model.
- Develop and field-test proposed measures and quality standards at
selected participating organizations.
- Produce a concise guidebook to assist libraries in assessing and
describing digital reference services in terms of specific measures
and quality standards.
Accomplishing these goals and objectives is essential if digital
reference services are to evolve successfully and be fully integrated as
part of library services.
Study Phases and Schedule
The study will be conducted over an eleven month period beginning
March 2001 (assuming adequate initial funding). The study will be
organized in a number of phases. Key tasks within these phases include:
Phase I: Project Organization. [Two months] In this phase, the
study team will develop detailed project tasking; review existing work
and resources related to evaluation of digital reference services and
related topics; establish a project Web site; and organize an advisory
committee for the project.
Phase II: Review of Best Practices from Participating Libraries.
[Two months] Members of the study team will conduct a selected number
of case studies/site visits with participating libraries that are
actively involved in digital reference service evaluation or that have
staff especially knowledgeable in this area.
Phase III: Developing and Field-Testing of Measures and Quality
Standards. [Four months] Based on the existing knowledge base, the
case studies, and the study team's knowledge, a set of proposed
measures and quality standards will be proposed. These will be
field-tested in a sample of participating library organizations.
Phase IV: Presentation of Findings and Final Report. [Two months]
The study team will present preliminary findings from the study at the
Third Annual VRD Digital Reference Conference in Atlanta in November
2001. Based on the feedback from this presentation and comments from
the advisory committee and others, the study team will then produce a
final report which will be a practical manual of assessment techniques
to produce measures and quality standards.
These tasks and the project schedule will be detailed during Phase I
and may be modified as the project progresses. Participating libraries
will receive a status report at the completion of each phase and will
receive the manual at the end of Phase IV.
Project Budget and Financial Arrangements
The proposed budget for this project is $125,000. This amount may be
modified depending on the level of participation and commitment the
project receives. The investigators will proceed with the project once
$100,000 in commitments are received. The primary expenditures in the
operating budget would be for personnel (65%); travel for site visits
and other meetings (20%); and the remainder for resources/communication.
Individuals and organizations can participate in the study at two
levels:
- Contributing Members will provide $5,000 to the project and will
have access to the project Web site of current activities and
receive regular project updates and status reports plus the
handbook.
- Sustaining Members will provide $10,000 or more to the project and
will have the benefits of contributing members plus they will be
able to serve on the project advisory committee and may be selected
for site visits and to field test measures and standards.
Potential sponsors should discuss their
involvement directly with Dr. Charles R. McClure or Dr. R. David Lankes.
Staffing and Organizational Capabilities
The research team for this project is drawn from leaders in digital
reference and assessment from the Information Institutes at Florida
State University and Syracuse University. Additional members of the
study team may be drawn from outside these two organizations.
Information Institute at Florida State University
The Information Use Management and Policy Institute at Florida State
University is directed by Dr. Charles R. McClure. Other staff to be
involved in the project include Dr. John Carlo Bertot and Bruce Fraser,
J.D. The Institute has been actively involved in a number of projects
related to assessment, evaluation, measurement, and quality standards in
the networked environment including projects funded by the U.S.
Institute for Museum and Library Services, the Association of Research
Libraries, and the Province of Alberta, Canada. Additional detail on
these and other activities at the Institute can be found at http://www.ii.fsu.edu/;
additional background and information regarding Charles R. McClure's
experience and research projects can be found at
http://slis-two.lis.fsu.edu/~cmcclure/.
Information Institute at Syracuse University
The Information Institute of Syracuse is a leading research center
for digital reference and part of Syracuse University's School of
Information Studies. The Institute created and operates the AskERIC
service (www.askeric.org) and the Virtual Reference Desk project
(www.vrd.org).
It has won numerous awards, answered over 200,000 digital reference
questions and runs the largest digital reference conference in the
country.
Research of the Institute has included creation of metadata standards
for resource discovery and digital reference and creating models of
building and maintaining digital reference services. Institute
researchers including Dr. R. David Lankes, Dr. Joanne Silverstein, Abby
Kasowitz, Blythe Bennett and Joann Wasik have also been engaged in
extensive research on Internet customer support in the federal context
and so-called AskA services. The Institute has produced numerous books
and articles on the topic of digital reference including AskA Starter
Kit: How to Build and Maintain Digital Reference Services, by R. David
Lankes and Abby S. Kasowitz; Building & Maintaining Internet
Information Services: K-12 Digital Reference Services, by R. David
Lankes; and "Quality Standards for Digital Reference
Consortia" by Abby Kasowitz, Blythe Bennett, and R. David Lankes in
Reference & User Services Quarterly, 39 (no. 4).
Additional information on the Information Institute of Syracuse is
available at: http://iis.syr.edu/
The Need for Action
Increasingly, digital reference services are being developed and
implemented at libraries across the country. Unfortunately, similar
emphasis on developing assessment techniques, measures, and standards is
not progressing at a similar rate. This study is crucial in creating a
series of standards and measures for developing quality digital
reference services.
By developing common understandings of quality and the costs of
quality, libraries can make informed decisions regarding digital
reference service. By understanding the nature of quality assessment for
digital reference, the library community can continue to provide
high-quality information service on the Web and set the standards for
providing expertise online. It is through this reference function that
we can continue to demonstrate that librarians offer more than
collections of links and data: they serve as essential human guides to
information resources and services for all information users.
Finally, if digital reference services are to evolve successfully as
bona fide library and information services, librarians need to engage in
ongoing assessment and evaluation of those services. Such assessment is
essential for the planning and development of these services, for cost
and financial decision-making, and perhaps most importantly, to ensure
that user information needs are met. This study is an important first
step in better understanding how digital library services can be
successfully integrated into existing library and information services.
References
Bertot, J. C.; McClure, C. R.; and Ryan, J. (2000). Statistics and
Performance Measures for Public Library Networked Services. Chicago:
American Library Association.
Hernon, Peter, and Whitman, John R. (2000). Delivering Satisfaction
and Service Quality: A Customer-Based Approach for Libraries. Chicago:
American Library Association.
Janes, J. (2000). Current Research in Digital Reference: Findings and
Implications. Presentation at Facets of Digital Reference, the VRD 2000
Annual Digital Reference Conference, 17 October, Seattle, WA.
Janes, J., and McClure, C.R. (1999). The Web as a Reference Tool:
Comparisons with Traditional Sources. Public Libraries, 38 (January
1999): 2-9.
Kasowitz, A., Bennett, B., and Lankes, R.D. (2000). Quality Standards
for Digital Reference Consortia. Reference & User Services
Quarterly, 39 (no. 4): 355-63.
LITA (1999). Top Tech Trends. [Online] Available: http://www.lita.org/committe/toptech/trendsmw99.htm
Virtual Reference Desk (2000). Facets of Quality for Digital
Reference Services. Version 4. [Online]. Available: http://www.vrd.org/training/facets10-00.htm
McClure/Lankes Assessing Quality in Digital Reference Services
DRAFT January 12, 2001
|
