Evaluation of the Digital Library Federation, 1995-2001. Summary report

Summary Report of the review panel as endorsed by the DLF Steering Committee

Submitted on September 28,2001 by the DLF review panel:
Bernard Hurley, University of California at Berkeley
Paul Mosher, University of Pennsylvania
Martin Runkle, University of Chicago
Gloriana St. Clair, Carnegie Mellon University
Sarah Thomas (Chair), Cornell University

As amended and unanimously approved by members of the DLF Steering Committee on November 14, 2001

In June 2001 the Digital Library Federation (DLF) Steering Committee approved the creation of a Review Panel to evaluate the progress of the DLF in achieving its goals and to consider the DLF's future.

What follows is the Panel's report as endorsed by the DLF Steering Committee at its meeting on November 14, 2001. The report explains why and how the evaluation was conducted and outlines its principle recommendations. It also supplies evidence submitted to the Review Panel in the course of its work.

Table of contents

Introduction

In June 2001 the Digital Library Federation (DLF) Steering Committee approved the creation of a Review Panel to evaluate the progress of the DLF in achieving its goals and to consider DLF's future. The founders of the National Digital Library Federation, DLF's precursor, had placed an initial five-year limit on the organization, and in its strategic plan approved in March 2000, the DLF Steering Committee had recommended that the DLF conduct an assessment before the conclusion of the DLF funding cycle in June 2002. The DLF Executive Committee formed a Review Panel to meet the charge of the Steering Committee.

The Charge

• Be undertaken by a Review Panel consisting of members of the DLF Steering Committee
• Assess progress to date and help shape any future program development;
• Consider whether, how, and under what organizational and financial terms the organization might be continued; and
• Result in a report from the Review Panel to the Steering Committee by 1 October 2001.

Methodology

The Review Panel identified key data and information it required as background to evaluate the DLF. At the Review Panel's request, the DLF Director, Dan Greenstein, prepared a review of DLF's progress meeting its aims, goals, and objectives; a description of DLF initiatives, their impacts, and the costs associated with them; and other documentation relating to DLF activity (Appendices 1 and 2). DLF engaged consultants to survey members and a select group of non-members using a list of questions prepared by the Review Panel ( Appendix 3). The Review Panel also interviewed the DLF Executive Director and the Council on Library and Information Resources (CLIR) President. The Chair of the Executive Committee was also consulted. The DLF Review Panel has created its report drawing on these rich and varied inputs.

General Findings and Recommendations

DLF members unanimously agreed that the Digital Library Federation has had a significant, positive impact on digital library development and that the DLF should continue its work. There was also strong consensus that the DLF benefits from periodic assessment.

Recommendation 1: The Digital Library Federation should continue for a limited term of five years (July 2002- June 2007) and should undergo a subsequent review in 2006.

To guide the DLF over the next five years, members called for the creation of a new strategic plan that would establish directions and priorities for its activities commensurate with available funding.

Recommendation 2: The Executive Committee should lead the Steering Committee through the development of a strategic plan.

Issue: Governance

While the DLF Member Survey revealed strong support for the current direction, productivity and leadership of DLF, it also reflected some confusion about the governance structure of DLF and its relationship to CLIR. Although informal structure was appropriate when the organization was young, the membership small, and the values and goals common among membership and staff, DLF may have grown faster than its institutional memory, and now is felt by the members to need a clear, open and stable governance structure understood by all.

Members would like the relationship between DLF and CLIR to be well-defined, have an open system for nominating and electing the Executive Committee, and be clear about the different functions of the Steering Committee and the Executive Committee, whose role most favor clarifying and strengthening. Members made clear that they take the term "federation" seriously: the Federation consists of member institutions who contribute substantially in funds and staff involvement. It is a participative, contributive organization rather than a membership or service organization. It is intended to be highly effective, have a smaller staff, a leaner organization, and depend on members' expert staff and third party agencies for program and project support rather than paid staff.

We urge that the following recommendations be incorporated in the DLF Bylaws, which should be drafted by the Executive Committee and passed by the Steering Committee.

Recommendation 3: DLF is governed by its Steering Committee, which consists of the senior executive of each contributing member or its designated representative. The DLF strategic plan, DLF program direction, membership criteria and issues, and dues and fees, are set by the Steering Committee upon the advice of the Executive Committee. The Steering Committee shall meet at least twice annually, as scheduled by the Executive Committee, to hear reports and transact the business of the Federation.

Recommendation 4: The Council on Library and Information Resources serves as the administrative home of DLF. The DLF Steering Committee exercises program and broad budgetary direction for DLF; CLIR has fiscal responsibility for DLF, but DLF funds are not intermingled with those of CLIR.

Recommendation 5: The DLF Executive Committee has authority for setting meeting agendas for DLF, and for meetings of the Steering Committee; it is responsible to the Steering Committee for DLF governance between meetings of the Steering Committee, and regularly meets between Steering Committee meetings. The Executive Committee should act as a bridge between the Steering Committee, the DLF Executive Director, and CLIR, as appropriate.

Recommendation 6: The Chair and members of the Executive Committee are elected by the Steering Committee from a slate of candidates nominated by a committee of members appointed by the Chair of the Executive Committee. There will be an opportunity for nominations from the floor at the time the slate of candidates is presented for election to the Steering Committee.

Recommendation 7: The DLF Executive Director is responsible to the DLF Steering Committee through its Executive Committee for the effective functioning of the Federation, and reports periodically to the Steering Committee on the condition and achievements of the Federation and its budget The DLF Executive Committee exercises programmatic oversight over the DLF Executive Director, while CLIR exercises administrative and fiscal oversight. The Executive Director also attends meetings of the CLIR Board, and reports periodically on the progress and accomplishments of DLF.

Issue: Organizational Effectiveness

The Steering Committee survey found DLF and its leaders uncommonly effective for a small, low-overhead organization. All members agreed they had profited from their investment in DLF. It is clear that DLF staff understand the goals and outcomes desired by members, and that DLF has enabled members to accomplish what could not have been done individually. DLF is fundamentally understood as a collaboratory focusing on digital library issues and projects of common concern. Frequently-cited examples of success in this role are the Open Archives Initiative (OAI) metadata harvesting study, access to grant fund support for key projects, identification of best practices and models, work on metadata standards, outcomes research, digital preservation, the Forums, and digital preservation.

There was much discussion of organizational size in the survey results. Consensus was that a size of 26-36 members is about right, and that the criteria for membership should emphasize participation and contribution. Genuine collaboration and participation are what set DLF apart as an organization--shared expertise was often cited as a major factor in DLF achievement. Useful outcomes rather than cachet should remain DLF's goal.

Recommendation 8: Steering Committee meetings should be thematic or emphasize issues of strategy, project direction, or policy. Reporting should be handled when appropriate by succinct written reports appended to the meeting agendas.

Issue: Communication

The Review Panel concluded that there were opportunities to strengthen communication between the Executive Director and the Executive Committee as well as with the Steering Committee. The DLF Executive Director should discuss proposed DLF initiatives with the Executive Committee, and calls for expressions of interest in participation in program activities. This procedure addresses concerns expressed during the survey that DLF members did not always learn of projects until they were underway or did not have a chance to join in them. DLF members, even with the aid of the DLF web page and regular emails, had difficulty in enumerating many DLF activities, suggesting that other forms of communication beyond the home page and emails may be desirable.

Recommendation 9: The DLF Executive Director and Executive Committee shall ensure that all DLF members learn of new initiatives through an open call for participation to DLF members, but this does not require appointing all who ask to serve.

The DLF has achieved considerable visibility for its accomplishments in its short lifetime. However, many non-DLF members remain unaware of the benefits of DLF or its contributions in advancing digital library developments. The DLF needs to continue to build credibility and influence.

Recommendation 10: The DLF members and Executive Director should make a concerted effort to publicize DLF activities, best practices, and strategic vision through presentations, publications, and other means.

The new set of publications initiated by DLF and CLIR responds to a need expressed by many for more effective dissemination of DLF outcomes and results.

Issue: Membership

Recommendation 11: The DLF may from time to time add new members who have the capacity and commitment to contribute substantial expertise to DLF's programs and initiatives and will advance the goals of the DLF membership.

Issue: Adequate funding for DLF Initiatives

Recommendation 12: The DLF Steering Committee should consider whether to continue to accept the CLIR contribution of in-kind services.

Recommendation 13: The DLF Steering Committee should vote on increasing dues by $1000 to an annual dues of $20,000.

Recommendation 14: The DLF Steering Committee should approve and encourage some use of the capital fund to pay for projects and initiatives that have previously been charged to the operations budget.

Issue: Broaden Member Participation

The strength of the DLF is in its membership and all members are expected to participate in its initiatives. The survey of DLF Steering Committee commissioned by this Review Committee identified three themes centered on member participation.

• The DLF Forums have been enormously successful.

Recommendation 15: The DLF Forums should continue.

Recommendation 16: The Review Committee recommends that member organizations which have not made presentations at the Forums be encouraged to do so, as learning of their efforts will benefit the entire Federation.

• Digital library project managers should have more opportunities to participate in DLF activities

The desire to increase DLF participation for staff with digital library management or development responsibilities was the most cited request identified by the survey.

Recommendation 17: The Review Committee recommends that the full Steering Committee discuss participation by digital library project managers in DLF activities and establish means to increase this level of participation.

• DLF working groups should have a broader member representation.

The survey identified a perception, if not a reality, that the same people are called upon to serve on working groups.

Recommendation 18: The Review Committee recommends that the DLF administration, working with the Executive Committee, broaden the pool of potential working group members. This could be accomplished by making the formation of working groups more visible, so qualified staff throughout the membership can volunteer and be considered for inclusion.

Appendix 1. Review of the Digital Library Federation's progress meeting its aims, goals, and objectives

Contents
A1.1. Introduction
A1.2. Aims
A1.3. Goals
A1.4. Objectives
A1.5. Directors' activity report for the period December 2000 - September 2001 (text omitted)
A1.6 Summary of DLF website use, January 2000 - July 2001 (text omitted)
A1.7. Level and extent of participation in DLF forums and working groups
A1.8. DLF publications and reports since inception

A1.1. Introduction

This document describes the aims, goals, and objectives of the DLF and its accomplishments in achieving them. It is prepared at the request of the DLF Review Panel and submitted to that Panel for its consideration as it evaluates the DLF's progress to date and its future possibilities.

The document makes explicit reference to the DLF Strategic Plan ( http://www.diglib.org/about/strategic.htm) that was agreed by the Steering Committee in March 2000, and assesses progress made by the DLF in meeting the goals and objectives set out in that plan.

Section A1.2. lists initiatives that the DLF has sponsored since its inception and includes initiatives that are planned but not yet formally begun. For each initiative, the list supplies a title, a brief description and assessment of impact where possible. A statement of the initiative's cost to the DLF was also supplied.

A1.2. Aims

The strategic plan states that the DLF leverages its members' reputations, investments, and research and development capacities to:

• Share and evaluate information about digital library tools, methods, practices, trends, and strategies
• Stimulate and share in the conduct of necessary digital library research and development
• Respond quickly and effectively to digital library challenges as they arise
• Exercise some influence for the library community over a rapidly changing information landscape
• Act as a catalyst in the development of innovative information services and organizations, and as an agent of learning for the profession
• Attract investment in essential digital library research and development activities
• Build a community of professionals appropriate to the development of digital libraries.

Progress in these areas is described below.

A1.2.1. Fostering shared evaluation of digital library tools, etc

• Publishing surveys of digital library developments at DLF members and other leading research libraries. Recent surveys have focused on strategies for developing sustainable scaleable digital collections; methods for assessing use of online collections and services; and the origins, focus, organization, funding, and policy environments surrounding leading digital library programs
• Ensuring that DLF-supported initiatives begin by evaluating libraries' needs and how they are currently being met at DLF member and other leading research libraries
• Regularly convening DLF forums
• Promoting specialized discussions on DLF-hosted listservs (e.g. on image quality, and on preservation of e-journals).

The impact of the DLF's work in this area may become evident from survey components of the DLF review ( Appendix 1, Section A1.3.).

A1.2.2. Stimulating and sharing in the conduct of necessary digital library research and development

The DLF has:

• Defined a stable metadata harvesting protocol attracting support for the protocol's maintenance, further refinement, and application from the National Science Foundation, the Coalition for Networked Information, The Andrew W. Mellon Foundation, and the British Library - see acknowledgements on the Open Archives website at http://www.openarchives.org
• Defined and encouraged adoption of schemes for representing technical, structural, and administrative metadata ("METS" is now maintained by the Library of Congress - see http://www.loc.gov/mets/)
• Defined and encouraged commercial adoption (e.g. by CrossRef) of localized reference-linking solutions
• Defined and encouraged development of and experimentation with certificate based authentication systems - examples of their use are available from Internet2 (http://middleware.internet2.edu/shibboleth/) and the Corporation for Research and Educational Networking (CREN - http://www.cren.net/)

A1.2.3. Responds quickly and effectively to digital library challenges as they arise

A1.2.4. Influence over for the library community

• Members' prestige and the profile of their visibly successful digital library programs
• Its exclusiveness as a small and selective membership organization
• The timeliness and quality of its initiatives
• Its network into initiatives, funding agencies, and other bodies that exercise some strategic influence over the development of digital libraries and digital library technologies
• The endorsement that the DLF gives to selected recommendations and activities

The DLF's communications program (see Appendix 2, Section A2.5) is essential to work in this area.

Evidence of progress is available in:

• Programs of the The Andrew W. Mellon Foundation that have built directly upon work conducted in DLF initiatives (e.g. the e-journals archiving program, ArtSTOR - see http://www.mellon.org/artstor%20announcement.html - and the Foundation's recent investment in metadata harvesting services)
• The guidance sought from bodies that have some strategic influence over how digital libraries develop [text deleted]
• Attention to and adoption of practices and models that are endorsed by the DLF (notably the Liblicense model license agreement between libraries and publishers, recommendations for the use of TEI guidelines in libraries, recommendations to CrossRef pertaining to the use of localized reference linking solutions)
• Extent to which the DLF Director is asked to represent the perspective of the DLF and its members (through public presentations, visits, and participation on advisory panels, etc.) and to bring these to bear on various digital library initiatives and programs
• Growing interest in membership in the DLF. Since January 2000, 2 libraries (at the Universities of Washington and Illinois at Urbana-Champaign) have joined the DLF and the National Archives intends to resume its membership. 11 institutions have expressed interest in joining and invited site visits [text deleted]. Additional preliminary discussions have been had with representatives of [text deleted].
• Growing use of the DLF website as evident in Appendix 1, Section A1.6.

A1.2.5. Acting as a catalyst in the development of innovative information services.

• Define and encourage investment in a distribution service (ArtSTOR - op cit) for digital images that support research, teaching, and learning in the visual arts and other humanities disciplines
• Define and attract investment in a cataloguing tool (ArtSTOR - op cit) that will support shared production of visual resource descriptions that meet some minimum level quality criteria
• Define minimal requirements for and attracted investment in planning e-journal archival repositories (op cit)
• Articulate the potential of and business cases for Internet gateway services built with the metadata harvesting protocol and attracted investment in such services from The Andrew W. Mellon Foundation, The National Science Foundation, the European Union Directorate General XIII, and the Joint Information Systems Committee of the UK's Higher Educational Funding Councils)
• Define and seek investment in a registry service to record information about digitally reformatted book and serial publications.

A1.2.6. Attracting investment in digital library research and development

DLF members leverage their membership in the DLF by attracting external funding for various digital library activities. The DLF Directors' Office has only a very partial knowledge of where and to what extent members are successful in this regard. More information may surface from survey components of the review. The Director's Office knows, for example, that DLF efforts have helped DLF members attract funding for:

• Work on e-journal archiving (from the Andrew W. Mellon Foundation)
• The development of OAI harvesting services (from the Andrew W. Mellon Foundation, the National Science Foundation)
• Practical experimentation with metadata repositories (from the Andrew W. Mellon Foundation and the National Science Foundation)
• Work on particular digitization and interoperability projects (e.g. from the Institute for Museum and Library Services, the National Endowment for the Humanities, The Andrew. W. Mellon Foundation, and the Getty Research Institute).

External funding for DLF initiatives flows through its administrative host, the Council on Library and Information Resources. The DLF has attracted funding for:

• Work developing a prototype and defining a business case for an image distribution service (from The Andrew W. Mellon Foundation - $100,000 for work on the Academic Image Cooperative
• Work on digital preservation (the DLF participates in a contract between the Library of Congress and CLIR through which CLIR will assist the Library of Congress in planning a national digital preservation program for which the Library of Congress will receive up to $100,000,000 from the US Congress. The DLF's role in this contract will be to help build a conceptual framework within which a national digital preservation strategy may be developed, and to define and implement the research agenda as appropriate for developing that framework. More information is available in Deanna Marcum, "CLIR Partners with LC on National Program for Digital Preservation". CLIR Issues, September/October 2001).

A1.2.7. Building a community of professionals

DLF members participate extensively and inclusively in DLF forums and working groups. In the meantime, the DLF publishes prolifically. Details about the level and extent of participation in DLF activities and about the DLF's publication profile are available in Appendix 1, sections A1.7. and A1.8., respectively.

Evidence of impacts may become available through the survey component of the DLF review (Appendix 3). Some evidence is available in evaluations that the Director's Office has conducted for the DLF forum, in particular. Those evaluations demonstrate that the forums offer an unparalleled opportunity for digital library professionals to learn and exchange information about cutting edge and highly practical research, development, and operational activities. Additional evidence is available in the extent to which new DLF initiatives are beginning to emerge from the casual discussions that take place at DLF forums and during DLF working group meetings (DLF initiatives arising from forum discussion include those on Open Source Software in libraries, on tools for developing Encoded Archival Descriptions, on technical metadata, on methods for assessing use of online collections and services). It is particularly encouraging to see some new ventures involving partners that had not hitherto worked together [text deleted].

A1.3. Goals

The DLF achieve its aims by working in areas of particular concern to its research library members. From March 2000, the DLF began working in six such areas, although it is expected that will change with time.

The goals the DLF set for its work in these areas is set out in language drawn from the strategic plan, alongside an indication of the progress the DLF has made achieving these goals.

A1.3.1. Digital library architectures, technologies, systems and tools

Since March 2000 we have not been as active in this area as one might have expected. This is not because there is no technical or architectural substance to our recent initiatives; rather because the technical components of DLF initiatives tend to be tied up in defining requirements for services that digital libraries require but cannot develop independently.

The following table supplies our stated goals, and the progress we have made in meeting them.

A1.3.2. Digital collections

A1.3.3. Use, users, user support, and user services

A1.3.4. Digital preservation

A1.3.5. Standards and best practices

A1.3.6. Institutional roles and responsibilities of the 21st-century digital library

A1.4. Objectives

The strategic plan indicates that the DLF uses the following means to meet its goals and realize its aims

• A central office including a director and a small support staff
• Distinguished Fellows - senior information professionals who are supported by the DLF to pursue research or lead catalytic initiatives in areas of mutual interest.
• An Executive committee that advises the Director on policy and other strategic matters
• A Steering Committee that acts as the governing body of the DLF
• Informal advisory groups that may be convened by the Director to advise on initiatives or progress within program areas
• DLF initiatives where shared investigative, information-sharing, and catalytic activities take place.
• A communications arm including:
  • a network of electronic mailing lists;
  • a quarterly newsletter which reports the progress of DLF initiatives to the members and through which, members report to one another on their own digital library developments;
  • online registries comprising information about (a) members' web-accessible public domain, digitized collections and (b) policies, strategies, working papers, standards and other application guidelines, and technical documentation developed by DLF members to inform or reflect upon their digital library development activities;
  • bi-annual DLF Forums; and
  • publications that report on DLF initiatives and on the proceedings of the DLF Forum.

With regard to the organizational means at the DLF's disposal, all work well and by and large as indicated in the strategic plan. Only the informal advisory groups have never really materialized, nor is a need for them felt at present.

Where the DLF's communications are is concerned, this is an essential means for building the community of professionals and inter-working institutions that is so essential to the success of individual DLF initiatives and to the organization over all.

In Spring 2001, the DLF Directors Office undertook a survey of how its various communications efforts were received by DLF members. The survey results are supplied in some greater detail in a companion document that lists and assesses individual DLF initiatives. In general, the survey found that the DLF's newsletter and its extensive and growing website are well received, and the DLF forum has evolved as both a major success and a leading digital library conference.

Details about DLF website use and publication profile are available in Appendix 1, Section A1.6.

More subjective evidence of the DLF's community-building progress is evident in the fact that the majority of new initiatives now originate outside the Director's Office.

A1.5. Summary of activity report of the DLF director for the period April 1 2000 - July 31 2001

Text omitted from public draft.

A1.6. Summary of DLF website use, January 2000 - July 2001

Text omitted from public draft.

A1.7. Level and extent of participation in DLF Forums and working groups

The following data are drawn from a database providing contact details about participants in DLF working groups, forums, and other events. The database includes basic information about participants in all DLF forums and in 19 of its working groups. A list of the working groups is supplied below.

Composition of the 19 DLF working groups for which participant information is available

Composition of DLF Forums

19 DLF Working Groups for which participation information is available

1. Academic Image Cooperative Advisory Board
2. Academic Image Cooperative Working Groups
3. The Nature and Use of the Scholarly Information Landscape
4. Assessing Image Quality
5. Developing Sustainable Digital Collections (advisory group) 1. group
6. Tools for Academic Electronic Publishing
7. Registry Services
8. Digital Library Programs' Organizational, Funding, and Policy Environments
9. METS
10. Open Archives Initiative (OA) Open Day (US based)
11. OAI Planning Meeting
12. OAI Steering Committee
13. OAI Technical Committee
14. Shared Cataloguing Tool for Visual Resources
15. TEI and XML in Digital Libraries
16. Reference Linking
17. Workshop on Social Science Data
18. Methods for Assessing Use of Online Collections and Services

A1.8. DLF publications and reports since inception

A1.8.1. Printed and online publications

The DLF has produced 8 printed publications since inception

The DLF has also produced 4 issues of the DLF Newsletter (online only), initiated in June 2000

A1.8.2. Progress reports on DLF initiatives (online only)

The DLF has produced over 40 online reports on its various initiatives. Reports are grouped below under the program area in which they appear.

A1.8.3. Architectures

• Open Archives Initiative
  • A New Approach to Finding Research Materials on the Web (July 2000) by Priscilla Caplan - describes the potential benefits to libraries and their users of the Open Archives Initiative and the Internet gateways that may be constructed with it
  • The Open Archives Initiative and Digital Libraries (January 2001) by Daniel Greenstein - introduces the OAI and supplies a brief history of the DLF's involvement with it as an organization
  • DLF Evaluation of the Open Arcihves Initiative (January 2001) by Daniel Greenstein - describes the work the DLF is undertaking in support of the development of a small number of Internet gateways through which users will access distributed digital library holdings as if they were part of a single uniform collection
• Tools for electronic publishing
  • Tools for Electronic Publishing (March 2001) by Maria Bonn - a discussion paper presented to a meeting of librarians and scholarly publishers to explore possible shared technical developments
• Reference linking
  • First Workshop on Linkage from Citations to Electronic Journal Literature. Report of a one day invitational workshop held on February 11, 1999 (March 1999) by staff at NISO
  • Report of the Second Workshop on Linkage from Citations to Journal Literature. Report of a one-day invitational workshop held on June 9, 1999 (July 1999) by staff at NISO
  • NISO/DLF/CrossRef Workshop on Localization in Reference Linking. Report of a one-day invitational workshop held on July 24, 2000 (August 2000) by staff at NISO
• Digital certificates
  • A Digital Library Authentication and Authorization Architecture (March 2000). Describing an architecture, protocol and operational model for using X.509 digital certificates for authentication and a directory service to serve user attributes to determine the level of authorized access to licensed online materials. The model was developed by the participants at a DLF-sponsored meeting January 19-20, 1999
  • Digital Library Authentication and Authorization (March 2000?). Describes the DLF initiative to develop an architecture, protocol and operational model for using X.509 digital certificates for authentication, etc.
  • Digital Certificate Infrastructure. Frequently Asked Questions (1999?). FAQ published jointly with the Corporation for Research and Educational Networking about the use of digital certificates, and targeted generally to senior campus administrators.
• Distributed finding aids
  • Supporting Access to Diverse and Distributed Finding Aids. A Final Reoprt to the Digital Library Federation on the Distributed Finding Aid Server Project (July 1999) by John Price Wilkin. Report on a project that explored the means and costs of searching encoded finding aids that are distributed at different institutions.

A1.8.4. Preservation

A1.8.5. Collections

• Registry of digital reproductions of paper-based books and serials
  • Registry of Digital Reproductions of Paper-based Books and Serials (July 2001) by Dale Flecker and Daniel Greenstein. Introduces the DLF's work developing a functional specification and business case for such a registry and facilitating its development
  • More Access at Less Cost: The Case for a Digital Registry (July 2001) by Gerald George. Making a case for and explaining the benefit to libraries of a registry of digital reproductions of paper-based books and serials.
  • Registry of Digital Reproductions of Paper-based Books and Serials. Functional requirements (July 2001) by Dale Flecker. Functional specification for a registry of digital reproductions of paper-based books and serials.
  • Draft report of a meeting held on 11 April 2001 to consider the potential uses of a service that registers digitized books and journals and to consider implementation (April 2001) by Daniel Greenstein.
• The Academic Image Cooperative
  • The Academic Image Cooperative (2000) by Daniel Greenstein. Introduces the DLF's work on this prototype service for aggregating and distributing curriculum-based digital images for courses in art history and other humanities disciplines.
  • Final Report of the AIC as submitted to the Andrew W. Mellon Foundation in August 2000 (August 2000) by Daniel Greenstein. Traces the progress of the AIC from its initial conception through the deveolopment of collection, technical, and image documentation standards, service specifications, and business plan.
  • The AIC Draft Collection Strategy and Development Framework (April 2000) by Daniel Greenstein. Resulting from a review of the AIC's progress during the period 1/99-2/00, the document outlines a collection strategy and development framework that may sustain the initiative in the longer term.
  • AIC Brochure (January 2000) by Rebecca Graham. This brief document outlines the AIC's vision and was developed to be circulated at conferences and other professional gatherings where the prototype was demonstrated
  • Academic Image Cooperative. Reports on initial meetings held January and May 1999 (June 1999).
• Shared cataloguing tool for visual resources
  • Towards a Shared Cataloguing Tool for VR Collections (December 2000) by Daniel Greenstein. Discussion paper used to launch an investigation into the functional requirements of and business case for a shared catalogue of or visual resource descriptions.
  • Draft report on a meeting to explore possibilities for developing a shared VR cataloguing service... (February 2001). Report on an initial meeting to consider the development of such a tool/service.
  • Shared Cataloguing Tool for Visual Resources (April 2001) by Max Marmor and Sherman Clarke. Report of a focus group meeting on the functional requirements developed by the DLF for a shared cataloguing tool for visual resources convened at ARLIS/NA Annual Conference, Getty Research Institute on April 1, 2001.
• Strategies for developing digital collections
  • Strategies for developing sustainable and scaleable digital library collections (May 2000) by Daniel Greenstein. Document outlining a DLF initiative that aims to assemble, review, and document practices adopted by libraries in developing their digital collections. The initiative resulted in several printed publications of the DLF.
• Social science data
  • Digital Library Federation Workshop on Social Science Data Archives (Feburary 1999). Reports on a meeting convened by the DLF on the state of the art of digital libraries in the social sciences to explore current problems and emerging solutions in three areas: facilities for users to discover and retrieve relevant and related data sets; means for users to interpret and evaluate the comparability of data sets; and tools for methods of data extraction and analysis.

A1.8.6. Standards and good practices

• Benchmarking digital reproductions of printed books and serials
  • Draft benchmark for digital reproductions of printed books and serial publications (July 2001) by Daniel Greenstein. Document recommending a minimum benchmark for digital reproductions of printed book and serial publications and outlining the importance, rationale, and implications of such a benchmark.
  • Report of a meeting of the DLF on preservation reformatting practices (July 2001) by Daniel Greenstein. Report of a meeting at which current practices were analyzed and benchmarks recommended
• Assessing image quality
  • Report of Imaging Practitioners Meeting on 30 March 2001 to Consider How the Quality of Digital Imaging Systems and Digital Images may be Fairly Evaluated (May 2001) by Stephen Chapman.
• Technical, administrative, and structural metadata
  • The Making of America, Part 2. Introducing work, supported by the DLF to organize and develop community practices for creating and encoding the digitized versions of primary sources and enabling readers to link seamlessly to these digitized surrogates directly from the finding aid descriptions of them. The MOA2 project received substantial implementation and other support from the National Endowment for the Humanities and the Andrew W. Mellon Foundation.
  • Structural, technical, and administrative metadata standards. A discussion document (December 2000) by Jerome McDonough. Document used to initiative discussion that led to the development of the METS standard.
  • Report of the Making of America II DTD DLF Workshop (March 2000), by Jerome McDonough. Report on a meeting convened at NYU in February 2001 to review experience with MOA2 DTD and to recommend changes. The report recommended the development of a new DTD, METS.
  • Metadata Encoding and Transmission Standard (METS). Official Web Site. Maintained by the Library of Congress, the website supplies overview, tutorial, schema and documentation for the use of the METS standard that developed out of a DLF-supported initiative.
• Licensing commercial content
  • Liblicense. Licensing Digital Information. A resource for Librarians. Web pages maintained at Yale University and including the CLIR/DLF Model License.
• TEI in Libraries
  • TEI Text Encoding in Libraries. Draft Guidelines for Best Encoding Practices. Version 1.0 (July 30, 1999) Perry Willetts. Guidelines growing out of a workshop convened to explore the use of the Text Encoding Initiative (TEI) and XML in libraries. They make recommendations pertaining to the application of the TEI Guidelines and particularly "best practices" for the encoding of electronic texts developed for different purposes. The guidelines have been endorsed and are in use by leading text centers in the US and Europe.
  • TEI and XML in Digital Libraries. Report of a Two-Day Meeting held June 30 - July 1, 1998 at the Library of Congress (1998) by Lee Ellen Friedland and John Price-Wilkin. Report on the meeting that framed work on the guidelines indicated above.
• Archival authority control
  • Developing a Standard for Recording Contextual Information for Archival and Manuscript Materials (December 1998). Report of a meeting at Yale University, New Haven, Connecticut (December 1998) to discuss the establishment of an international collaborative project to advance the definition and implementation of a methodology for recording information about the contexts in which archival records, personal papers, and similar materials have been created and used

A1.8.7. Use and users

• Methods for assessing use and usability of online collections and services
  • Usage, Usability, and User support (April 200) by Daniel Greenstein. Report of a discussion convened at the DLF Forum on 2 April 2000 to frame a DLF investigation into methods for assessing the use and usability of online collections and services
• Assessing changing patterns of library use
  • How and Why Libraries are Changing (January 2001) by Denise Troll. A discussion paper used to focus a DLF initiative to investigate the extent and use of the scholarly information landscape at universities and liberal arts colleges, and the library's contributions to that landscape.
  • Dimension and Use of the Scholarly Information Environment (July 2001), by Lynn Dagar, Daniel Greenstein, LeighWatson Healy. Proposal outlining aims, goals, methods for the research that is indicated above.

A1.8.8. Library roles and responsibilities

• Digital library policies, organizations, and practices
  • Draft report of a meeting held on 10 April in Washington DC to discuss preliminary results of a survey issued by the DLF to its members (May 2001) by D Greenstein, S Thorin, D Mckinney. Report on a meeting to discuss results of a survey that was used to help identify the institutional contexts in which digital libraries are being developed and to create a profile of the programs and initiatives at institutions that comprise the Digital Library Federation (DLF).
  • Digital Library Policies, Organizations, and Practices. DLF Survey (January 2001) by Daniel Greenstein and Suzanne Thorin. Survey instrument used by the DLF to help identify the institutional contexts in which digital libraries are being developed and to create a profile of the programs and initiatives at institutions that comprise the Digital Library Federation (DLF).

Appendix 2. DLF initiatives: description, impacts, and costs

Table of contents

A2.1. Introduction

From its inception in 1995, the DLF set itself the task of "creating the means to bring together digital materials - from across the nation and beyond - that will be made accessible to students, scholars, citizens everywhere". As it enters its sixth year, it is settling on a process for achieving this ambitious aim. In particular, the DLF is demonstrating its ability in:

• incubating new services and tools that digital libraries commonly require but cannot independently develop or sustain;
• identifying, endorsing, and promoting the standards and good and best practices that support the digital library's interests in high-quality, interoperable, and sustainable online collections and services;
• pooling members' research and development effort for common benefit; and
• facilitating communication amongst DLF members and ensuring that knowledge gained by the DLF is transferred into the broader community.

This document lists initiatives that the DLF has sponsored since its inception and includes initiatives that are planned but not yet formally begun. [Cost indications have been omitted from the public draft].

Initiatives are grouped with reference to the DLF's aims as:

• Catalytic (work incubating services etc)
• Work on standards and best practice
• Pooled research
• Communications

A2.2. Catalytic efforts

DLF funding stimulates the development of so-called infrastructural or utility tools and services that are commonly required by digital libraries but beyond their independent means. Work in this area develops functional specifications and business cases for the development of these tools or services. Working in this way, the DLF seeks to reduce the risk to investors who have the financial capacity to develop such tools and services. In this respect, DLF investment may be seen as incentive or venture funding.

A2.3. Standards and best practices

Leveraging its members' collective influence the DLF identifies, endorses, and promotes those standards and good practices that support the digital library's interests in high-quality, interoperable, persistent, and sustainable online collections and services. It is also developing the network of strategic alliances it will need to ensure that its good and best practice recommendations are maintained and communicated to the broadest possible community.

Although there is an opportunistic element in the way that priorities are assigned to work in this area, effort increasingly grows out of the best practice requirements that surface in developing functional and business cases for essential utility tools and services (see Appendix 2, Section A2.2.).

A2.4. Pooled research

The DLF pools research and development expertise at member and other institutions to survey and evaluate existing practice and to answer pressing strategic, organizational, and technical questions. Work in this area is driven by members' perceived needs, by opportunities that are available to conduct meaningful research, and, increasingly, by the questions arising from DLF investigations into infrastructural tools and services (see Appendix 2, Section A2.2)

A2.5. Communications

To build a community of professionals, the DLF facilitates its members' communication, collaboration, and cross-fertilization. Early indications from an evaluation instrument recently circulated to staff at member institutions suggest the DLF's newsletter (now published twice rather than four times a year) and its extensive and growing website are well received.

Appendix 3. Digital Library Federation Review. Survey of members and other decision makers. Executive Summary

September 3, 2001

Conducted by: The Communications Office, Inc.
108 East Del Ray Avenue
Alexandria, Virginia 22301
(703) 567-4977

Revised by the DLF Review Panel to Protect Confidentiality

Table of Contents

A3.1. Survey of DLF Steering Committee: Findings and Highlights

1. Why are you a member of the DLF?
   The theme for many was "collaboration." There appears to be an overarching philosophy that the sum of the work of the DLF is greater than its parts. Members are eager, especially for their staff members, to share information and problem solving. With the tremendous volume of work to be done, there needs to be economy of effort among members. The majority of respondents see their institutions as contributing strong currency to the collaborative process that will benefit the "whole." A few talk about what DLF can provide to their individual institutions. It is important to members that the DLF's work fill a unique niche and not be redundant in areas addressed by other organizations. Many Steering Committee members talked about the DLF's value to their staffs' development.
2. Has the DLF enabled its members to accomplish what could not have been done individually or informally?
   The unanimous response to this question is "yes."
3. What has the DLF enabled its members to accomplish?
   Once again, the opportunity to collaborate was a central theme. More specifically, the Open Archives Initiative metadata harvesting study and facilitated access to grant money were often mentioned. Among the other responses were cross reference mobilization, survey research, identification of best practices and models, metadata standards, preservation of electronic journals, registry of digitized books and serials, the academic image cooperative (ArtSTOR), and standards work such as METS.
4. Has your institution profited from its investment of time, money, and attention in the DLF? What value has it received?
   Every institutional member said yes, they have profited from their investment in the DLF. Again, they noted the value of collaboration and facilitated access to grant money. There is no doubt that the DLF forums are highly valued. At this point in the survey, a few comments emerged indicating that some Steering Committee members believe the collaborative value of DLF rests more with university staff members who are directly involved in digital programs, not with the Steering Committee members. This theme recurs more strongly further into the survey.
5. How can the DLF better assist you in leveraging your investment?
   The spectrum of responses was broad, as indicated in the following sample responses. Once again, there were comments urging more inclusion, in terms of library staff from DLF institutions. Members want DLF to take a stronger, more visible leadership role on digital issues. The belief is that with strength will come more funding opportunities for the organization and its members. They would like to see their investment in the DLF reap more grant dollars.
6. Has your rationale for belonging to the DLF changed over the years? If so, how?
   About half of the respondents said their rationale for belonging to the DLF has changed. Many said their expectations for DLF have changed positively as the organization has matured. Once again, the subject of giving library staff a more active role in the DLF was discussed by a few. There appears to have been a shift in mindset as members initially set out to create one digital library; whereas today, they are working on a collaborative model built on multiple institutional contributions. There is concern expressed by a few that, as DLF has grown, its power and energy have been diluted by members who do not have the passion and expertise upon which the organization thrived initially.
7. What has the DLF accomplished as an organization?
   Many people listed specific accomplishments, such as a registry for digital objects, reference linking, standards for digitization, archiving, best practices, and work user studies, among others. Providing opportunities for collaboration, bringing visibility to digital libraries and their issues, and attracting funders were more broadly defined accomplishments. Yet, others felt accomplishments were still pending and that more emphasis should be placed on outcomes.
8. In what areas, if any, has the DLF developed a leadership role?
   Members view leadership from many perspectives. Some were quick to point out specific initiatives-- digitization, open archives initiative, metadata standards and metadata harvesting, and most notably, best practices. Others judged DLF's leadership status by prominence in government and funding agency meetings. Still others believe its leadership is best characterized by its catalytic ability to inspire collaboration and novel thinking. One member took exception to the term "leadership," preferring to cite "success" in providing positive outcomes to benefit scholars.
9. Should the DLF continue?
   The answer was a resounding "yes." One member said "maybe," adding "I think it needs regular reviews. If it is not accomplishing stuff, it should go out of business."
10. Should the DLF operate at its current level of funding?
    There was no consensus. Some members expressed a desire to better understand DLF's future direction before they could provide a definitive answer. Others approached the question from their institution's ability to pay higher dues, as opposed to assessing the best interest of the DLF. Of that group, some said they would not be willing to pay more; a few said they would be happy to pay more if the increase reaps greater benefits; still others thought an increase might be warranted, but it should be absorbed through additional members. Also, it is interesting to note that some equated an increase in funding with a larger organization, as opposed to expanded project funding.
11. What funding sources should support the DLF?
    Members-to varying degrees-believe the current membership dues structure, combined with foundation and government grants, is appropriate. Other suggestions included commercial sponsorships and creating an entrepreneurial consulting venture for DLF.
12. Should the DLF remain a limited-term organization, or should it establish itself as a permanent organization?
    Most of the respondents believe DLF should remain a limited-term organization.
13. What is the optimal size for the DLF?
    There was no consensus on optimal size. In distilling the content of the various responses, it would be accurate to say most members believe the current size is manageable (although a few feel that 26 is too large). However, there are those who feel the DLF should be more inclusive and open membership to an unlimited number of institutions that are able to meet the organization's high standards and dues requirements. These particular respondents are not necessarily advocating for a large DLF membership; they believe that the high entrance standards and financial commitment will effectively regulate the size of the membership.

    Others were more concerned about the quality than the actual number of members. This topic was introduced earlier in the survey.

    This question elicited talk about the potential of changing organizational structure to accommodate the present membership and, definitely, a larger one. That topic is addressed in greater detail further in the survey.

14. To whom should the DLF expand its membership?
    [Text deleted]
15. What institutions, if any, should DLF expand its membership?
    [Text deleted]
16. Is the DLF's current organizational and governing structure effective in meeting the work and leadership requirements of the organization?
    About half of the respondents believe DLF's current organizational and governing structure is effective. However, many of them provided a qualified "yes." The creation of an Executive Committee to make quick decisions and keep the organization nimble has been welcomed. There are concerns about the function of the Steering Committee. Some say it is ineffective because of its size; others say it is ineffective because its members are only peripherally involved in the day-to-day, hands-on operation of digital library programs. The theme of engaging more project people in the DLF emerges again.
17. What improvements, if any, might you suggest in DLF's current organizational and governing structure?
    [Text deleted.]

    The Steering Committee continues to be a focus of concern for some, in terms of its personality and effectiveness. There is a feeling that some of the library directors do not have the expertise or interest to stay actively engaged. This points back to the contention by some that the Steering Committee members should address DLF administrative and broad programmatic issues; those with specific digital library expertise should engage one another and spearhead collaborative efforts.

18. What has been the most successful part of the DLF agenda?
    As one can see from the following comments, members see a number of DLF successes. The overwhelming response is the DLF forums, followed by the Open Archives Initiative. It is obvious, reviewing these responses, that members find opportunities to engage their staffs in the work of DLF to be highly desirous.
19. What has been the least successful part of the DLF agenda?
    Once again, issues of communication and Steering Committee dynamics were discussed. Other topics were mentioned-limited publication dissemination, lack of strategic planning, disconnects between endorsements and implementation, borne-digital vs. retrospective priorities, funding brokering, etc. However, relative to the comments provided in the rest of the survey, this "least successful" question yielded a limited number of answers.
20. What might the future emphasis of the DLF be?
    Members have such a diverse "wish list" for DLF that it is not possible to identify particular trends. The responses speak for themselves.
21. Does the DLF need other collaborators to achieve its goals?
    Most respondents see the necessity for collaborators, but they are quick to point out that the DLF membership should remain as it is. They do not see collaborators as future DLF members. A few members are willing to explore collaborative relationships with commercial vendors, but a general distrust of that group was expressed. Yet, others express a desire to at least stay abreast of commercial developments that could benefit the DLF.

    [Text deleted.]

A3.2. Survey of Key Informants: Findings and Highlights

1. Have DLF initiatives advanced the work of the digital library community?
   All respondents answered "yes."
2. How has the DLF advanced the digital library community's work?
   Most respondents answered in the broad sense of providing a forum for collaboration, setting standards, conceptualizing and articulating necessary agendas, promoting staff development, infrastructure development, etc.

   [Text deleted.]

3. Has any DLF work influenced how you've developed your digital library program?
   This question was not relevant to all respondents. University librarians cited influence in the area of standards, infrastructure issues, and shared developments and approaches. [Text deleted.]
4. Has the DLF enabled 'the community' to accomplish what it could not do individually or informally?
   The consensus was "yes," but with qualifying statements:

   [Text deleted.]

5. Should the DLF continue?
   Of the eight respondents, only five gave an unqualified "yes."

   [Text deleted.]

Who is The Communications Office?

Digital Information issues are not new to The Communications Office. It recently conducted a similar survey for the Council on Library and Information Resources (CLIR). The firm counts among its clients the Library of Congress, Congressional Research, and National Archives & Records Administration. It wrote the Digital Library Federation's organizational brochure ("Chaos and Revolution") under the direction of Don Waters as well as the collateral materials for CLIR's "Into The Future" digital memory crisis campaign in 1998.

return to top >>