Usage, Usability and User Support

Report of a discussion group convened at the DLF Forum on 2 April 2000

Dan Greenstein and Denise Troll
Version 1.1
26 May 2000

Participants

I. Purposes of the meeting

• To identify a research, development, and information sharing agenda for the DLF that will inform and support its members in their efforts to evaluate the use and usability of digital library collections and services.
• To identify user communities and provide support for their use of digital collections and services.
• To recommend next steps that may be taken to develop the identified agenda.

II. Research development and information sharing agendas identified

The group identified three broad areas that might benefit from some shared effort:

1. Research methodologies and their application in the digital library context

• Identify, evaluate, and determine the potential shared application of quantitative and qualitative research methods appropriate for evaluating the nature, extent, quality and effectiveness of the use and usability of digital collections and services.
• Encourage commercial vendors and local developers to apply whatever learning results from use and usability studies in the design and development of digital library collections, services, and applications.
• Encourage application of whatever learning results from use and usability studies in the development of user support services and in professional development activities (e.g. for public service librarians)

2. User support in a 24/7 digital library

• Identify communities that use physical as well as digital libraries.
• In part through an environmental scan of current practice, identify, evaluate, and determine the potential shared investigation of methods appropriate for delivering public service functions and user support in a 24/7 digital library service environment.
• Contribute to the development of performance measures, best practices and knowledge management appropriate to 24/7 user support services.

3. The library as space

• Study the use of the library as both a physical and virtual place.
• Identify social interactions in the physical library that are not currently available in the digital library, determine which social interactions are essential to the mission and values of a library and investigate ways to support these interactions in the digital library environment.

These three areas are intrinsically inter-related. The research methodologies developed to evaluate use and usability of digital collections and services (area 1) will inform strategies for supplying user support in the 24/7 digital library (area 2) and decisions about the utilization of library space (area 3). Similarly investigations into the use of library space (area 3) and, for example, the importance of encouraging social interactions, will inform the development of 24/7 support services (area 2) where such interactions may also be encouraged albeit in an online environment.

Despite the inter-relatedness of the three areas, the participants felt that the DLF should focus initially in a single area. Priority was given to research methodologies and their application to analyses of the nature, extent, quality, and effectiveness of use of digital library collections and services (area 1).

Given the work currently being conducted by ICOLC and other organizations to develop measures appropriate to assessing use of commercial data resources, the DLF initiative will concentrate on measures appropriate to assessing the use of non-commercial collections and services.

Clearly, any research results arising from such a program would need to be assessed in light of their implications for user support services and the allocation and use of library space.

III. Next steps

The group outlined a six-step implementation process for work in the first of these three areas, i.e., methods for evaluating and enhancing the use and usability of digital library collections and services. (The same steps might apply to work undertaken in the other two areas.)

Step 1. Prepare a problem statement addressing the question "why do libraries need to analyze the use and usability of digital collections and services?"

Draft problem statement: Libraries need to assess the use and usability of digital collections and services to do or improve:

• Capacity planning (e.g. networking, hardware and software)
• Systems design (e.g. the appearance and functionality of systems that deliver digital collections and services)
• User support (e.g., the design and organization of support services)
• Collection development (e.g. the acquisition of third-party commercial content and services; selection of locally managed content for digitization)
• Strategic planning (e.g., the library's institutional role, organization, funding and developmental direction)

Although both qualitative and quantitative methods exist and are being deployed to meet some of these assessment needs, there is substantial room for shared activity to help libraries:

• Articulate more precisely the kinds of data required to support their assessment needs
• Identify, evaluate and share information about appropriate methods, tools and techniques and how they may be applied to gather these data
• Develop comparable datasets and analyses that reduce redundant effort while enhancing a collective understanding of user needs, expectations and behaviors
• Advocate amongst third-party content and service providers to supply relevant data that meet the library's needs
• Learn from and build upon relevant experience and expertise that exists outside the library sector

Step 2. Identify specific assessment needs, the kinds of data that are being generated to meet them, and the methods being used to generate and analyze those data.

Initially the process will involve limited telephone survey involving as respondents professionals at DLF member institutions who are currently involved in assessing use and usability of non-commercial digital library collections and services.

In conducting this survey, the DLF will be looking for respondents who have experience in applying any of the assessment methods listed in column a below to any of the collections or services that are listed in column b.

Respondents will be asked to discuss what they hope to learn from their assessments, the data they are generating, how they are generating and analyzing those data, and how they are using the results they are obtaining. They will also be asked for any methodological documentation that describes their efforts and for reports based in whole or in part on the results they are obtaining.

Step 3. Develop a bibliography of relevant "publications" which discuss relevant assessment methods and identify and involve any interested library organizations (e.g. ICOLC, ARL) and individuals from outside the library community who have relevant expertise.

Step 4. Develop an inventory of current research practices at DLF member sites and elsewhere, both within and outside the library community.

Development of the inventory may rely upon two modes of data collection:

• Extending the survey indicated above in step2 to a broader community of respondents;
• Commissioned research into assessment methods and their application within and outside the digital library arena.

Step 5. Conduct evaluation and gap analysis.

Convene a workshop of experts to evaluate the effectiveness of research methodologies inventoried in step 4 in meeting specific assessment needs. The outcome of the workshop will identify where assessment needs are and are not being met with existing research methods.

Step 6. Initiate further work, as appropriate.

Based on the outcome of the meeting in Step 5:

1. Where assessment needs are not being met with existing research methods, initiate appropriately focused research and development work to devise methods to meet these needs.
2. Where assessment needs are more or less being met with existing research methods, explore prospects for sharing the research results generated with these methods.

return to top >>