random quote Link: Publications Forum Link: About DLF Link: News
Link: Digital Collections Link: Digital Production Link: Digital Preservation Link: Use, users, and user support Link: Build: Digital Library Architectures, Systems, and Tools
photo of books






Please send the DLF Director your comments or suggestions.

Evaluation of the Digital Library Federation, 1995-2001. Summary report

Summary Report of the review panel as endorsed by the DLF Steering Committee

Submitted on September 28,2001 by the DLF review panel:
Bernard Hurley, University of California at Berkeley
Paul Mosher, University of Pennsylvania
Martin Runkle, University of Chicago
Gloriana St. Clair, Carnegie Mellon University
Sarah Thomas (Chair), Cornell University

As amended and unanimously approved by members of the DLF Steering Committee on November 14, 2001

In June 2001 the Digital Library Federation (DLF) Steering Committee approved the creation of a Review Panel to evaluate the progress of the DLF in achieving its goals and to consider the DLF's future.

What follows is the Panel's report as endorsed by the DLF Steering Committee at its meeting on November 14, 2001. The report explains why and how the evaluation was conducted and outlines its principle recommendations. It also supplies evidence submitted to the Review Panel in the course of its work.

Table of contents

General Findings and Recommendations
Appendix 1. Review of the Digital Library Federation's progress meeting its aims, goals, and objectives
Appendix 2. DLF initiatives: description, impacts, and costs
Appendix 3. Digital Library Federation Review. Survey of members and other decision makers. Executive Summary


In June 2001 the Digital Library Federation (DLF) Steering Committee approved the creation of a Review Panel to evaluate the progress of the DLF in achieving its goals and to consider DLF's future. The founders of the National Digital Library Federation, DLF's precursor, had placed an initial five-year limit on the organization, and in its strategic plan approved in March 2000, the DLF Steering Committee had recommended that the DLF conduct an assessment before the conclusion of the DLF funding cycle in June 2002. The DLF Executive Committee formed a Review Panel to meet the charge of the Steering Committee.

The Charge

The charge noted that the DLF evaluation should:
  • Be undertaken by a Review Panel consisting of members of the DLF Steering Committee
  • Assess progress to date and help shape any future program development;
  • Consider whether, how, and under what organizational and financial terms the organization might be continued; and
  • Result in a report from the Review Panel to the Steering Committee by 1 October 2001.


The Review Panel identified key data and information it required as background to evaluate the DLF. At the Review Panel's request, the DLF Director, Dan Greenstein, prepared a review of DLF's progress meeting its aims, goals, and objectives; a description of DLF initiatives, their impacts, and the costs associated with them; and other documentation relating to DLF activity (Appendices 1 and 2). DLF engaged consultants to survey members and a select group of non-members using a list of questions prepared by the Review Panel ( Appendix 3). The Review Panel also interviewed the DLF Executive Director and the Council on Library and Information Resources (CLIR) President. The Chair of the Executive Committee was also consulted. The DLF Review Panel has created its report drawing on these rich and varied inputs.

General Findings and Recommendations

DLF members unanimously agreed that the Digital Library Federation has had a significant, positive impact on digital library development and that the DLF should continue its work. There was also strong consensus that the DLF benefits from periodic assessment.

Recommendation 1: The Digital Library Federation should continue for a limited term of five years (July 2002- June 2007) and should undergo a subsequent review in 2006.

To guide the DLF over the next five years, members called for the creation of a new strategic plan that would establish directions and priorities for its activities commensurate with available funding.

Recommendation 2: The Executive Committee should lead the Steering Committee through the development of a strategic plan.

Issue: Governance

While the DLF Member Survey revealed strong support for the current direction, productivity and leadership of DLF, it also reflected some confusion about the governance structure of DLF and its relationship to CLIR. Although informal structure was appropriate when the organization was young, the membership small, and the values and goals common among membership and staff, DLF may have grown faster than its institutional memory, and now is felt by the members to need a clear, open and stable governance structure understood by all.

Members would like the relationship between DLF and CLIR to be well-defined, have an open system for nominating and electing the Executive Committee, and be clear about the different functions of the Steering Committee and the Executive Committee, whose role most favor clarifying and strengthening. Members made clear that they take the term "federation" seriously: the Federation consists of member institutions who contribute substantially in funds and staff involvement. It is a participative, contributive organization rather than a membership or service organization. It is intended to be highly effective, have a smaller staff, a leaner organization, and depend on members' expert staff and third party agencies for program and project support rather than paid staff.

We urge that the following recommendations be incorporated in the DLF Bylaws, which should be drafted by the Executive Committee and passed by the Steering Committee.

Recommendation 3: DLF is governed by its Steering Committee, which consists of the senior executive of each contributing member or its designated representative. The DLF strategic plan, DLF program direction, membership criteria and issues, and dues and fees, are set by the Steering Committee upon the advice of the Executive Committee. The Steering Committee shall meet at least twice annually, as scheduled by the Executive Committee, to hear reports and transact the business of the Federation.

Recommendation 4: The Council on Library and Information Resources serves as the administrative home of DLF. The DLF Steering Committee exercises program and broad budgetary direction for DLF; CLIR has fiscal responsibility for DLF, but DLF funds are not intermingled with those of CLIR.

Recommendation 5: The DLF Executive Committee has authority for setting meeting agendas for DLF, and for meetings of the Steering Committee; it is responsible to the Steering Committee for DLF governance between meetings of the Steering Committee, and regularly meets between Steering Committee meetings. The Executive Committee should act as a bridge between the Steering Committee, the DLF Executive Director, and CLIR, as appropriate.

Recommendation 6: The Chair and members of the Executive Committee are elected by the Steering Committee from a slate of candidates nominated by a committee of members appointed by the Chair of the Executive Committee. There will be an opportunity for nominations from the floor at the time the slate of candidates is presented for election to the Steering Committee.

Recommendation 7: The DLF Executive Director is responsible to the DLF Steering Committee through its Executive Committee for the effective functioning of the Federation, and reports periodically to the Steering Committee on the condition and achievements of the Federation and its budget The DLF Executive Committee exercises programmatic oversight over the DLF Executive Director, while CLIR exercises administrative and fiscal oversight. The Executive Director also attends meetings of the CLIR Board, and reports periodically on the progress and accomplishments of DLF.

Issue: Organizational Effectiveness

The Steering Committee survey found DLF and its leaders uncommonly effective for a small, low-overhead organization. All members agreed they had profited from their investment in DLF. It is clear that DLF staff understand the goals and outcomes desired by members, and that DLF has enabled members to accomplish what could not have been done individually. DLF is fundamentally understood as a collaboratory focusing on digital library issues and projects of common concern. Frequently-cited examples of success in this role are the Open Archives Initiative (OAI) metadata harvesting study, access to grant fund support for key projects, identification of best practices and models, work on metadata standards, outcomes research, digital preservation, the Forums, and digital preservation.

There was much discussion of organizational size in the survey results. Consensus was that a size of 26-36 members is about right, and that the criteria for membership should emphasize participation and contribution. Genuine collaboration and participation are what set DLF apart as an organization--shared expertise was often cited as a major factor in DLF achievement. Useful outcomes rather than cachet should remain DLF's goal.

Recommendation 8: Steering Committee meetings should be thematic or emphasize issues of strategy, project direction, or policy. Reporting should be handled when appropriate by succinct written reports appended to the meeting agendas.

Issue: Communication

The Review Panel concluded that there were opportunities to strengthen communication between the Executive Director and the Executive Committee as well as with the Steering Committee. The DLF Executive Director should discuss proposed DLF initiatives with the Executive Committee, and calls for expressions of interest in participation in program activities. This procedure addresses concerns expressed during the survey that DLF members did not always learn of projects until they were underway or did not have a chance to join in them. DLF members, even with the aid of the DLF web page and regular emails, had difficulty in enumerating many DLF activities, suggesting that other forms of communication beyond the home page and emails may be desirable.

Recommendation 9: The DLF Executive Director and Executive Committee shall ensure that all DLF members learn of new initiatives through an open call for participation to DLF members, but this does not require appointing all who ask to serve.

The DLF has achieved considerable visibility for its accomplishments in its short lifetime. However, many non-DLF members remain unaware of the benefits of DLF or its contributions in advancing digital library developments. The DLF needs to continue to build credibility and influence.

Recommendation 10: The DLF members and Executive Director should make a concerted effort to publicize DLF activities, best practices, and strategic vision through presentations, publications, and other means.

The new set of publications initiated by DLF and CLIR responds to a need expressed by many for more effective dissemination of DLF outcomes and results.

Issue: Membership

Recommendation 11: The DLF may from time to time add new members who have the capacity and commitment to contribute substantial expertise to DLF's programs and initiatives and will advance the goals of the DLF membership.

Issue: Adequate funding for DLF Initiatives

[Text deleted]

Recommendation 12: The DLF Steering Committee should consider whether to continue to accept the CLIR contribution of in-kind services.

Recommendation 13: The DLF Steering Committee should vote on increasing dues by $1000 to an annual dues of $20,000.

Recommendation 14: The DLF Steering Committee should approve and encourage some use of the capital fund to pay for projects and initiatives that have previously been charged to the operations budget.

Issue: Broaden Member Participation

The strength of the DLF is in its membership and all members are expected to participate in its initiatives. The survey of DLF Steering Committee commissioned by this Review Committee identified three themes centered on member participation.

  • The DLF Forums have been enormously successful.

Recommendation 15: The DLF Forums should continue.

Recommendation 16: The Review Committee recommends that member organizations which have not made presentations at the Forums be encouraged to do so, as learning of their efforts will benefit the entire Federation.

  • Digital library project managers should have more opportunities to participate in DLF activities

The desire to increase DLF participation for staff with digital library management or development responsibilities was the most cited request identified by the survey.

Recommendation 17: The Review Committee recommends that the full Steering Committee discuss participation by digital library project managers in DLF activities and establish means to increase this level of participation.

  • DLF working groups should have a broader member representation.

The survey identified a perception, if not a reality, that the same people are called upon to serve on working groups.

Recommendation 18: The Review Committee recommends that the DLF administration, working with the Executive Committee, broaden the pool of potential working group members. This could be accomplished by making the formation of working groups more visible, so qualified staff throughout the membership can volunteer and be considered for inclusion.

Appendix 1. Review of the Digital Library Federation's progress meeting its aims, goals, and objectives

Submitted to the DLF Review Panel for its consideration by
Daniel Greenstein
DLF Director

A1.1. Introduction
A1.2. Aims
A1.3. Goals
A1.4. Objectives
A1.5. Directors' activity report for the period December 2000 - September 2001 (text omitted)
A1.6 Summary of DLF website use, January 2000 - July 2001 (text omitted)
A1.7. Level and extent of participation in DLF forums and working groups
A1.8. DLF publications and reports since inception

A1.1. Introduction

This document describes the aims, goals, and objectives of the DLF and its accomplishments in achieving them. It is prepared at the request of the DLF Review Panel and submitted to that Panel for its consideration as it evaluates the DLF's progress to date and its future possibilities.

The document makes explicit reference to the DLF Strategic Plan ( http://www.diglib.org/about/strategic.htm) that was agreed by the Steering Committee in March 2000, and assesses progress made by the DLF in meeting the goals and objectives set out in that plan.

Section A1.2. lists initiatives that the DLF has sponsored since its inception and includes initiatives that are planned but not yet formally begun. For each initiative, the list supplies a title, a brief description and assessment of impact where possible. A statement of the initiative's cost to the DLF was also supplied.

A1.2. Aims

The strategic plan states that the DLF leverages its members' reputations, investments, and research and development capacities to:

  • Share and evaluate information about digital library tools, methods, practices, trends, and strategies
  • Stimulate and share in the conduct of necessary digital library research and development
  • Respond quickly and effectively to digital library challenges as they arise
  • Exercise some influence for the library community over a rapidly changing information landscape
  • Act as a catalyst in the development of innovative information services and organizations, and as an agent of learning for the profession
  • Attract investment in essential digital library research and development activities
  • Build a community of professionals appropriate to the development of digital libraries.

Progress in these areas is described below.

A1.2.1. Fostering shared evaluation of digital library tools, etc

. The DLF has done this by:
  • Publishing surveys of digital library developments at DLF members and other leading research libraries. Recent surveys have focused on strategies for developing sustainable scaleable digital collections; methods for assessing use of online collections and services; and the origins, focus, organization, funding, and policy environments surrounding leading digital library programs
  • Ensuring that DLF-supported initiatives begin by evaluating libraries' needs and how they are currently being met at DLF member and other leading research libraries
  • Regularly convening DLF forums
  • Promoting specialized discussions on DLF-hosted listservs (e.g. on image quality, and on preservation of e-journals).

The impact of the DLF's work in this area may become evident from survey components of the DLF review ( Appendix 1, Section A1.3.).

A1.2.2. Stimulating and sharing in the conduct of necessary digital library research and development

The DLF does this by helping to define essential research and development agenda, align partners as required to carry out those agenda, and assist those partners in attracting the support necessary to fulfill their objectives. Evidence of the DLF's progress in this area is available here and in Appendix 1, section A1.2.5.

The DLF has:

  • Defined a stable metadata harvesting protocol attracting support for the protocol's maintenance, further refinement, and application from the National Science Foundation, the Coalition for Networked Information, The Andrew W. Mellon Foundation, and the British Library - see acknowledgements on the Open Archives website at http://www.openarchives.org
  • Defined and encouraged adoption of schemes for representing technical, structural, and administrative metadata ("METS" is now maintained by the Library of Congress - see http://www.loc.gov/mets/)
  • Defined and encouraged commercial adoption (e.g. by CrossRef) of localized reference-linking solutions
  • Defined and encouraged development of and experimentation with certificate based authentication systems - examples of their use are available from Internet2 (http://middleware.internet2.edu/shibboleth/) and the Corporation for Research and Educational Networking (CREN - http://www.cren.net/)

A1.2.3. Responds quickly and effectively to digital library challenges as they arise

Through directors and staff at DLF member institutions, the DLF stays abreast of pressing digital library issues. With its lightweight organization, its nimble decision-making and funding apparatuses, and its access to expertise in member institutions, the DLF is able to respond quickly to issues that are particularly pressing and to take advantage of opportunities as they arise. Progress in this area and evidence of the DLF's nimbleness is evident in work that has been undertaken quickly in response to need and opportunity as it was identified, e.g. with localized reference linking, certificate based authentication, registries for digitally reformatted materials, etc.

A1.2.4. Influence over for the library community

The DLF seeks to exercise its influence over the broader community by capitalizing on:
  • Members' prestige and the profile of their visibly successful digital library programs
  • Its exclusiveness as a small and selective membership organization
  • The timeliness and quality of its initiatives
  • Its network into initiatives, funding agencies, and other bodies that exercise some strategic influence over the development of digital libraries and digital library technologies
  • The endorsement that the DLF gives to selected recommendations and activities

The DLF's communications program (see Appendix 2, Section A2.5) is essential to work in this area.

Evidence of progress is available in:

  • Programs of the The Andrew W. Mellon Foundation that have built directly upon work conducted in DLF initiatives (e.g. the e-journals archiving program, ArtSTOR - see http://www.mellon.org/artstor%20announcement.html - and the Foundation's recent investment in metadata harvesting services)
  • The guidance sought from bodies that have some strategic influence over how digital libraries develop [text deleted]
  • Attention to and adoption of practices and models that are endorsed by the DLF (notably the Liblicense model license agreement between libraries and publishers, recommendations for the use of TEI guidelines in libraries, recommendations to CrossRef pertaining to the use of localized reference linking solutions)
  • Extent to which the DLF Director is asked to represent the perspective of the DLF and its members (through public presentations, visits, and participation on advisory panels, etc.) and to bring these to bear on various digital library initiatives and programs
  • Growing interest in membership in the DLF. Since January 2000, 2 libraries (at the Universities of Washington and Illinois at Urbana-Champaign) have joined the DLF and the National Archives intends to resume its membership. 11 institutions have expressed interest in joining and invited site visits [text deleted]. Additional preliminary discussions have been had with representatives of [text deleted].
  • Growing use of the DLF website as evident in Appendix 1, Section A1.6.

A1.2.5. Acting as a catalyst in the development of innovative information services.

Here the DLF helps to define requirements and business cases for such services, and to align partners and attract investment as necessary to develop them. Evidence of the DLF's progress in this area is available here and under Aim 2. The DLF has helped to:
  • Define and encourage investment in a distribution service (ArtSTOR - op cit) for digital images that support research, teaching, and learning in the visual arts and other humanities disciplines
  • Define and attract investment in a cataloguing tool (ArtSTOR - op cit) that will support shared production of visual resource descriptions that meet some minimum level quality criteria
  • Define minimal requirements for and attracted investment in planning e-journal archival repositories (op cit)
  • Articulate the potential of and business cases for Internet gateway services built with the metadata harvesting protocol and attracted investment in such services from The Andrew W. Mellon Foundation, The National Science Foundation, the European Union Directorate General XIII, and the Joint Information Systems Committee of the UK's Higher Educational Funding Councils)
  • Define and seek investment in a registry service to record information about digitally reformatted book and serial publications.

A1.2.6. Attracting investment in digital library research and development

The DLF has attracted funding for member institutions for the Director's office. The DLF's support for work in various areas has also encouraged more general investment flowing from a variety of agencies into non-DLF member institutions. No attempt will be made to assess progress in leveraging funding of this latter type.

DLF members leverage their membership in the DLF by attracting external funding for various digital library activities. The DLF Directors' Office has only a very partial knowledge of where and to what extent members are successful in this regard. More information may surface from survey components of the review. The Director's Office knows, for example, that DLF efforts have helped DLF members attract funding for:

  • Work on e-journal archiving (from the Andrew W. Mellon Foundation)
  • The development of OAI harvesting services (from the Andrew W. Mellon Foundation, the National Science Foundation)
  • Practical experimentation with metadata repositories (from the Andrew W. Mellon Foundation and the National Science Foundation)
  • Work on particular digitization and interoperability projects (e.g. from the Institute for Museum and Library Services, the National Endowment for the Humanities, The Andrew. W. Mellon Foundation, and the Getty Research Institute).

External funding for DLF initiatives flows through its administrative host, the Council on Library and Information Resources. The DLF has attracted funding for:

  • Work developing a prototype and defining a business case for an image distribution service (from The Andrew W. Mellon Foundation - $100,000 for work on the Academic Image Cooperative
  • Work on digital preservation (the DLF participates in a contract between the Library of Congress and CLIR through which CLIR will assist the Library of Congress in planning a national digital preservation program for which the Library of Congress will receive up to $100,000,000 from the US Congress. The DLF's role in this contract will be to help build a conceptual framework within which a national digital preservation strategy may be developed, and to define and implement the research agenda as appropriate for developing that framework. More information is available in Deanna Marcum, "CLIR Partners with LC on National Program for Digital Preservation". CLIR Issues, September/October 2001).

A1.2.7. Building a community of professionals

Here the DLF has attempted to encourage a culture of collaboration amongst staff at member institutions, and to provide venues for their fruitful exchange of ideas and experiences. Forums (the DLF has hosted 4 since Fall 1999), the physical meetings and off-line communication of DLF-supported working groups (the DLF has supported over 30 such groups), and DLF publications and progress reports (the DLF has produced 8 printed publications since inception, 4 online Newsletters since July 2000, and over 40 online reports, again since inception) are crucial components of this effort.

DLF members participate extensively and inclusively in DLF forums and working groups. In the meantime, the DLF publishes prolifically. Details about the level and extent of participation in DLF activities and about the DLF's publication profile are available in Appendix 1, sections A1.7. and A1.8., respectively.

Evidence of impacts may become available through the survey component of the DLF review (Appendix 3). Some evidence is available in evaluations that the Director's Office has conducted for the DLF forum, in particular. Those evaluations demonstrate that the forums offer an unparalleled opportunity for digital library professionals to learn and exchange information about cutting edge and highly practical research, development, and operational activities. Additional evidence is available in the extent to which new DLF initiatives are beginning to emerge from the casual discussions that take place at DLF forums and during DLF working group meetings (DLF initiatives arising from forum discussion include those on Open Source Software in libraries, on tools for developing Encoded Archival Descriptions, on technical metadata, on methods for assessing use of online collections and services). It is particularly encouraging to see some new ventures involving partners that had not hitherto worked together [text deleted].

A1.3. Goals

The DLF achieve its aims by working in areas of particular concern to its research library members. From March 2000, the DLF began working in six such areas, although it is expected that will change with time.

The goals the DLF set for its work in these areas is set out in language drawn from the strategic plan, alongside an indication of the progress the DLF has made achieving these goals.

A1.3.1. Digital library architectures, technologies, systems and tools

Since March 2000 we have not been as active in this area as one might have expected. This is not because there is no technical or architectural substance to our recent initiatives; rather because the technical components of DLF initiatives tend to be tied up in defining requirements for services that digital libraries require but cannot develop independently.

The following table supplies our stated goals, and the progress we have made in meeting them.

Goal Progress
Define, clarify, and develop prototypes for digital library systems and system components. Evidence of progress in well recognized and increasingly accepted work on key architectural components of certificate-based authentication, localized reference linking; architecture of digital repositories; on the OAI Metadata Harvesting protocol; and on the functional requirements of infrastructural digital library services such as a registry for digital reproductions of books and serials, an image distribution service, etc.
Scan the larger technical environment for potentially important trends and practices. Not formally active, though most DLF initiatives respond to such trends.
Encourage technology transfer and information sharing between and among DLF members and between the DLF and appropriate commercial and industrial sectors. Information exchange amongst members happens best at the Forum, in DLF initiatives and working groups, and through Newsletter and the web-based reports on DLF initiatives. Work on localized reference linking and on OAI may potentially transfer between libraries and industry.
Communicate technical directions and accomplishments of the DLF to a wider audience. Not a central concern though the DLF website and DLF publications are publicly accessible and DLF staff and fellows regularly speak to audiences of non-DLF members and publish in relevant journals etc.

A1.3.2. Digital collections

Goal Progress
Identify, evaluate and, develop collection strategies and practices that are appropriate for the digital library and assess the legal, organizational, and business implications of these strategies. The DLF has commissioned research and published extensively in this area, notably in its series "Strategies for building sustainable digital collections", but also through work on AIC and elsewhere.
Prepare guidelines and other informational materials that help inform local collection development decision. The above publications include such guidelines. In addition, DLF has developed model licenses, and benchmarks for the production and documentation of certain kinds of digital content.
Encourage the development of new kinds of scholarly collections that take full advantage of computer and network technologies Work on AIC/ArtSTOR focused on collections of digital images. Work with OAI encourages development of Internet gateways. There is a much more to be done encouraging digital libraries to gain maximum advantage from other kinds of digital objects such as GIS, social science data, audio, film, etc.

A1.3.3. Use, users, user support, and user services

Goal Progress
Engage more effectively with user communities as a means of building better, more useful, and more useable collections and services. Work on collection strategies (Appendix 1, Section A1.3.2.) and on methods for assessing use of online collections and services (Appendix 1, Section A1.3.3.) emphasize the importance of library engagement with user communities. DLF initiatives do not typically involve end users except insofar as professional library staff use the ideas, tools, services, etc. that DLF initiatives focus on.
Provide data about users and their behavior in online environments as a means of informing the development of digital library collections and services. The DLF is helping to extend the amount and improve the quality of such data focusing in particular on data that are comparable across institutions. Notable contributions have come through our study of the information-seeking behavior of humanities scholars (Palmer and Brockman); our investigation (with Outsell Inc.) into the nature and use of the scholarly information landscape at liberal arts colleges and research universities (forthcoming). We are also working on assessment methods, notably by documenting effective practices currently in use through our survey of effective methods for assessing use of online collections and services. An instructional workshop based on the survey was held in June 2001 and attended by nearly 45 people.
Build sustainable user support services that encourage use and are appropriate to our evolving digital library service environments. There is more to be done here. At present our work in this area extends to support for an investigation by Charles McClure and David Lankes into benchmarking online reference services.

A1.3.4. Digital preservation

Goal Progress
Encourage initiatives that result in some practical experience of digital preservation. Work defining and building cross-community consensus around minimum requirements of e-journal archives contributed to The Andrew W. Mellon Foundation' e-journal archiving program. The DLF continues to support the program.

The DLF is also supporting CLIR in its work with the Library of Congress in framing the national digital preservation program for which the Library has received up to $100,000,000 from the US Congress. The DLF's role in this contract will be to help build a conceptual framework within which a national digital preservation strategy may be developed.

Develop and work within a framework that allows such initiatives to document, compare, and evaluate their experiences. The DLF maintains the website for the Mellon's e-journal archiving program and supports its participants in information sharing and reporting to the community. It will also play a role in helping to ensure that the learning involved in work with the Library of Congress is communicated to the broader community.
Continuing research and development in the most poorly understood areas associated with digital preservation. Work under the contract with the Library of Congress and the Mellon e-journal archiving program promises to move us into new areas including the preservation of electronic books, the Web, digital audio, etc.

A1.3.5. Standards and best practices

Goal Progress
Encourage greater interoperability and exchange of digital collections. The DLF has helped make interoperability and information exchange more feasible than it has been at any time in the past. Here one thinks about the DLF's work developing and encouraging widespread adoption or use of tools and services that fundamentally lower barriers to interoperability. The tools include minimum-level agreements (benchmarks) pertaining to the use of various data and metadata standards; network protocols (e.g. OAI, X.509). Services include an image distribution service (ArtSTOR), a registry for digitally reformatted books and serials, Internet gateways built with the OAI harvesting protocol.

There is a great deal more to be done with both services and tools. But perhaps the greatest effort remains in non-technical realms.

  • We lack a common understanding of what "interoperability" actually entails, or even what it means
  • Accordingly, we lack the means of "benchmarking" interoperable collections and aggregating services
  • Institutions are not yet fully comfortable contributing information content to aggregated collections with which they may not be obviously associated
  • Beyond the large bibliographic databases, we lack services - even prototypes - that clearly demonstrate the advantages that cross-collection searching provides to end-users and to information providers such as libraries, or that can provide information on how users want to engage with such services.
Facilitating long-term access to such collections. As above.
Exercising some influence over the behavior of third-party data suppliers We have had some influence here by organizationally endorsing particular practices. We have been particularly successful with localized reference linking solutions, with recommended uses of TEI in libraries, and with the Liblicense model licensing agreement. We have high hopes for OAI, benchmarks for digitally reformatted books and serial publications, and our technical metadata standards (METS).

A1.3.6. Institutional roles and responsibilities of the 21st-century digital library

Goal Progress
Undertake a survey that provides baseline data about the organizational, financial, and legal contexts in which digital libraries are beginning to emerge. The survey has been completed and an initial report is available. Case studies are being developed to enrich that report and will be included in the final publication (January 2002).
Develop papers targeting university presidents and provosts of universities and liberal arts colleges, documenting key digital library challenges and demonstrating their relevance to university mission. With CLIR we have created CLIRinghouse - a periodic newsletter that has this aim precisely.
Identify other communities that shape the landscapes in which digital libraries develop, assess their interests and motivations, and prepare a targeted literature about the digital library's significance and its issues. CLIRinghouse has this growth potential.

A1.4. Objectives

The strategic plan indicates that the DLF uses the following means to meet its goals and realize its aims

  • A central office including a director and a small support staff
  • Distinguished Fellows - senior information professionals who are supported by the DLF to pursue research or lead catalytic initiatives in areas of mutual interest.
  • An Executive committee that advises the Director on policy and other strategic matters
  • A Steering Committee that acts as the governing body of the DLF
  • Informal advisory groups that may be convened by the Director to advise on initiatives or progress within program areas
  • DLF initiatives where shared investigative, information-sharing, and catalytic activities take place.
  • A communications arm including:
    • a network of electronic mailing lists;
    • a quarterly newsletter which reports the progress of DLF initiatives to the members and through which, members report to one another on their own digital library developments;
    • online registries comprising information about (a) members' web-accessible public domain, digitized collections and (b) policies, strategies, working papers, standards and other application guidelines, and technical documentation developed by DLF members to inform or reflect upon their digital library development activities;
    • bi-annual DLF Forums; and
    • publications that report on DLF initiatives and on the proceedings of the DLF Forum.

With regard to the organizational means at the DLF's disposal, all work well and by and large as indicated in the strategic plan. Only the informal advisory groups have never really materialized, nor is a need for them felt at present.

Where the DLF's communications are is concerned, this is an essential means for building the community of professionals and inter-working institutions that is so essential to the success of individual DLF initiatives and to the organization over all.

In Spring 2001, the DLF Directors Office undertook a survey of how its various communications efforts were received by DLF members. The survey results are supplied in some greater detail in a companion document that lists and assesses individual DLF initiatives. In general, the survey found that the DLF's newsletter and its extensive and growing website are well received, and the DLF forum has evolved as both a major success and a leading digital library conference.

Details about DLF website use and publication profile are available in Appendix 1, Section A1.6.

More subjective evidence of the DLF's community-building progress is evident in the fact that the majority of new initiatives now originate outside the Director's Office.

A1.5. Summary of activity report of the DLF director for the period April 1 2000 - July 31 2001

Text omitted from public draft.

A1.6. Summary of DLF website use, January 2000 - July 2001

Text omitted from public draft.

A1.7. Level and extent of participation in DLF Forums and working groups

The following data are drawn from a database providing contact details about participants in DLF working groups, forums, and other events. The database includes basic information about participants in all DLF forums and in 19 of its working groups. A list of the working groups is supplied below.

Composition of the 19 DLF working groups for which participant information is available

Number of participants in working groups 406
Number of distinct individuals 340
Number of participants from DLF member institutions 221
Number of distinct individuals from DLF institutions 177

Composition of DLF Forums

Number of participants in 4 DLF forums 291
Number of individuals who have attended 4 DLF Forums 209

19 DLF Working Groups for which participation information is available

  1. Academic Image Cooperative Advisory Board
  2. Academic Image Cooperative Working Groups
  3. The Nature and Use of the Scholarly Information Landscape
  4. Assessing Image Quality
  5. Developing Sustainable Digital Collections (advisory group) 1. group
  6. Tools for Academic Electronic Publishing
  7. Registry Services
  8. Digital Library Programs' Organizational, Funding, and Policy Environments
  9. METS
  10. Open Archives Initiative (OA) Open Day (US based)
  11. OAI Planning Meeting
  12. OAI Steering Committee
  13. OAI Technical Committee
  14. Shared Cataloguing Tool for Visual Resources
  15. TEI and XML in Digital Libraries
  16. Reference Linking
  17. Workshop on Social Science Data
  18. Methods for Assessing Use of Online Collections and Services

A1.8. DLF publications and reports since inception

A1.8.1. Printed and online publications

The DLF has produced 8 printed publications since inception

Strategies for Building Digitized Collections (September 2001) by Abby Smith

Selection and Presentation of Commercially Available Electronic Resources: Issues and Practices (August 2001) by Tim Jewell

Building Sustainable Collections of Free Third-Party Web Resources (June 2001) by Louis Pitschmann

Guides to Quality in Visual Resource Imaging (July 2000)

Systems of Knowledge Organization for Digital Libraries: Beyond Traditional Authority Files (April 2000) by Gail Hodge

The Making of America II Testbed Project: A Digital Library Service Model (December 1999) by Bernard J. Hurley, John Price-Wilkin, Merrilee Proffitt and Howard Besser

Preserving the Whole: A Two-Track Approach to Rescuing Social Science Data and Metadata (June 1999) by Ann Green with JoAnn Dionne and Martin Dennis

Enabling Access: A Report on a Workshop on Access Management (February 1999) by Caroline Arms with Judith Klavans and Don Waters

The DLF has also produced 4 issues of the DLF Newsletter (online only), initiated in June 2000

A1.8.2. Progress reports on DLF initiatives (online only)

The DLF has produced over 40 online reports on its various initiatives. Reports are grouped below under the program area in which they appear.

A1.8.3. Architectures

  • Open Archives Initiative
    • A New Approach to Finding Research Materials on the Web (July 2000) by Priscilla Caplan - describes the potential benefits to libraries and their users of the Open Archives Initiative and the Internet gateways that may be constructed with it
    • The Open Archives Initiative and Digital Libraries (January 2001) by Daniel Greenstein - introduces the OAI and supplies a brief history of the DLF's involvement with it as an organization
    • DLF Evaluation of the Open Arcihves Initiative (January 2001) by Daniel Greenstein - describes the work the DLF is undertaking in support of the development of a small number of Internet gateways through which users will access distributed digital library holdings as if they were part of a single uniform collection
  • Tools for electronic publishing
    • Tools for Electronic Publishing (March 2001) by Maria Bonn - a discussion paper presented to a meeting of librarians and scholarly publishers to explore possible shared technical developments
  • Reference linking
    • First Workshop on Linkage from Citations to Electronic Journal Literature. Report of a one day invitational workshop held on February 11, 1999 (March 1999) by staff at NISO
    • Report of the Second Workshop on Linkage from Citations to Journal Literature. Report of a one-day invitational workshop held on June 9, 1999 (July 1999) by staff at NISO
    • NISO/DLF/CrossRef Workshop on Localization in Reference Linking. Report of a one-day invitational workshop held on July 24, 2000 (August 2000) by staff at NISO
  • Digital certificates
    • A Digital Library Authentication and Authorization Architecture (March 2000). Describing an architecture, protocol and operational model for using X.509 digital certificates for authentication and a directory service to serve user attributes to determine the level of authorized access to licensed online materials. The model was developed by the participants at a DLF-sponsored meeting January 19-20, 1999
    • Digital Library Authentication and Authorization (March 2000?). Describes the DLF initiative to develop an architecture, protocol and operational model for using X.509 digital certificates for authentication, etc.
    • Digital Certificate Infrastructure. Frequently Asked Questions (1999?). FAQ published jointly with the Corporation for Research and Educational Networking about the use of digital certificates, and targeted generally to senior campus administrators.
  • Distributed finding aids
    • Supporting Access to Diverse and Distributed Finding Aids. A Final Reoprt to the Digital Library Federation on the Distributed Finding Aid Server Project (July 1999) by John Price Wilkin. Report on a project that explored the means and costs of searching encoded finding aids that are distributed at different institutions.

A1.8.4. Preservation

Preservation of electronic scholarly journals
  • Minimum criteria for an archival repository of digital scholarly journals, version 1.2 (May 2000) by Daniel Greenstein and Deanna Marcum. Outlines minimum criteria agreed at invitational workshops of librarians and publishers respectively. The criteria serve as a touchstone for the repositories being planned by projects involved in The Andrew W. Mellon Foundation's e-journal archiving program
  • Website of The Andrew W. Mellon Foundation's e-journal archiving program (from February 2001). Numerous web-documents introducing the program and describing its aims and giving details about the individual projects including their successful grant applications and periodically updated progress reports.

A1.8.5. Collections

  • Registry of digital reproductions of paper-based books and serials
    • Registry of Digital Reproductions of Paper-based Books and Serials (July 2001) by Dale Flecker and Daniel Greenstein. Introduces the DLF's work developing a functional specification and business case for such a registry and facilitating its development
    • More Access at Less Cost: The Case for a Digital Registry (July 2001) by Gerald George. Making a case for and explaining the benefit to libraries of a registry of digital reproductions of paper-based books and serials.
    • Registry of Digital Reproductions of Paper-based Books and Serials. Functional requirements (July 2001) by Dale Flecker. Functional specification for a registry of digital reproductions of paper-based books and serials.
    • Draft report of a meeting held on 11 April 2001 to consider the potential uses of a service that registers digitized books and journals and to consider implementation (April 2001) by Daniel Greenstein.
  • The Academic Image Cooperative
    • The Academic Image Cooperative (2000) by Daniel Greenstein. Introduces the DLF's work on this prototype service for aggregating and distributing curriculum-based digital images for courses in art history and other humanities disciplines.
    • Final Report of the AIC as submitted to the Andrew W. Mellon Foundation in August 2000 (August 2000) by Daniel Greenstein. Traces the progress of the AIC from its initial conception through the deveolopment of collection, technical, and image documentation standards, service specifications, and business plan.
    • The AIC Draft Collection Strategy and Development Framework (April 2000) by Daniel Greenstein. Resulting from a review of the AIC's progress during the period 1/99-2/00, the document outlines a collection strategy and development framework that may sustain the initiative in the longer term.
    • AIC Brochure (January 2000) by Rebecca Graham. This brief document outlines the AIC's vision and was developed to be circulated at conferences and other professional gatherings where the prototype was demonstrated
    • Academic Image Cooperative. Reports on initial meetings held January and May 1999 (June 1999).
  • Shared cataloguing tool for visual resources
    • Towards a Shared Cataloguing Tool for VR Collections (December 2000) by Daniel Greenstein. Discussion paper used to launch an investigation into the functional requirements of and business case for a shared catalogue of or visual resource descriptions.
    • Draft report on a meeting to explore possibilities for developing a shared VR cataloguing service... (February 2001). Report on an initial meeting to consider the development of such a tool/service.
    • Shared Cataloguing Tool for Visual Resources (April 2001) by Max Marmor and Sherman Clarke. Report of a focus group meeting on the functional requirements developed by the DLF for a shared cataloguing tool for visual resources convened at ARLIS/NA Annual Conference, Getty Research Institute on April 1, 2001.
  • Strategies for developing digital collections
    • Strategies for developing sustainable and scaleable digital library collections (May 2000) by Daniel Greenstein. Document outlining a DLF initiative that aims to assemble, review, and document practices adopted by libraries in developing their digital collections. The initiative resulted in several printed publications of the DLF.
  • Social science data
    • Digital Library Federation Workshop on Social Science Data Archives (Feburary 1999). Reports on a meeting convened by the DLF on the state of the art of digital libraries in the social sciences to explore current problems and emerging solutions in three areas: facilities for users to discover and retrieve relevant and related data sets; means for users to interpret and evaluate the comparability of data sets; and tools for methods of data extraction and analysis.

A1.8.6. Standards and good practices

  • Benchmarking digital reproductions of printed books and serials
    • Draft benchmark for digital reproductions of printed books and serial publications (July 2001) by Daniel Greenstein. Document recommending a minimum benchmark for digital reproductions of printed book and serial publications and outlining the importance, rationale, and implications of such a benchmark.
    • Report of a meeting of the DLF on preservation reformatting practices (July 2001) by Daniel Greenstein. Report of a meeting at which current practices were analyzed and benchmarks recommended
  • Assessing image quality
    • Report of Imaging Practitioners Meeting on 30 March 2001 to Consider How the Quality of Digital Imaging Systems and Digital Images may be Fairly Evaluated (May 2001) by Stephen Chapman.
  • Technical, administrative, and structural metadata
    • The Making of America, Part 2. Introducing work, supported by the DLF to organize and develop community practices for creating and encoding the digitized versions of primary sources and enabling readers to link seamlessly to these digitized surrogates directly from the finding aid descriptions of them. The MOA2 project received substantial implementation and other support from the National Endowment for the Humanities and the Andrew W. Mellon Foundation.
    • Structural, technical, and administrative metadata standards. A discussion document (December 2000) by Jerome McDonough. Document used to initiative discussion that led to the development of the METS standard.
    • Report of the Making of America II DTD DLF Workshop (March 2000), by Jerome McDonough. Report on a meeting convened at NYU in February 2001 to review experience with MOA2 DTD and to recommend changes. The report recommended the development of a new DTD, METS.
    • Metadata Encoding and Transmission Standard (METS). Official Web Site. Maintained by the Library of Congress, the website supplies overview, tutorial, schema and documentation for the use of the METS standard that developed out of a DLF-supported initiative.
  • Licensing commercial content
    • Liblicense. Licensing Digital Information. A resource for Librarians. Web pages maintained at Yale University and including the CLIR/DLF Model License.
  • TEI in Libraries
    • TEI Text Encoding in Libraries. Draft Guidelines for Best Encoding Practices. Version 1.0 (July 30, 1999) Perry Willetts. Guidelines growing out of a workshop convened to explore the use of the Text Encoding Initiative (TEI) and XML in libraries. They make recommendations pertaining to the application of the TEI Guidelines and particularly "best practices" for the encoding of electronic texts developed for different purposes. The guidelines have been endorsed and are in use by leading text centers in the US and Europe.
    • TEI and XML in Digital Libraries. Report of a Two-Day Meeting held June 30 - July 1, 1998 at the Library of Congress (1998) by Lee Ellen Friedland and John Price-Wilkin. Report on the meeting that framed work on the guidelines indicated above.
  • Archival authority control
    • Developing a Standard for Recording Contextual Information for Archival and Manuscript Materials (December 1998). Report of a meeting at Yale University, New Haven, Connecticut (December 1998) to discuss the establishment of an international collaborative project to advance the definition and implementation of a methodology for recording information about the contexts in which archival records, personal papers, and similar materials have been created and used

A1.8.7. Use and users

  • Methods for assessing use and usability of online collections and services
    • Usage, Usability, and User support (April 200) by Daniel Greenstein. Report of a discussion convened at the DLF Forum on 2 April 2000 to frame a DLF investigation into methods for assessing the use and usability of online collections and services
  • Assessing changing patterns of library use
    • How and Why Libraries are Changing (January 2001) by Denise Troll. A discussion paper used to focus a DLF initiative to investigate the extent and use of the scholarly information landscape at universities and liberal arts colleges, and the library's contributions to that landscape.
    • Dimension and Use of the Scholarly Information Environment (July 2001), by Lynn Dagar, Daniel Greenstein, LeighWatson Healy. Proposal outlining aims, goals, methods for the research that is indicated above.

A1.8.8. Library roles and responsibilities

  • Digital library policies, organizations, and practices
    • Draft report of a meeting held on 10 April in Washington DC to discuss preliminary results of a survey issued by the DLF to its members (May 2001) by D Greenstein, S Thorin, D Mckinney. Report on a meeting to discuss results of a survey that was used to help identify the institutional contexts in which digital libraries are being developed and to create a profile of the programs and initiatives at institutions that comprise the Digital Library Federation (DLF).
    • Digital Library Policies, Organizations, and Practices. DLF Survey (January 2001) by Daniel Greenstein and Suzanne Thorin. Survey instrument used by the DLF to help identify the institutional contexts in which digital libraries are being developed and to create a profile of the programs and initiatives at institutions that comprise the Digital Library Federation (DLF).

Appendix 2. DLF initiatives: description, impacts, and costs

Prepared for the DLF's five-year review by
Daniel Greenstein
DLF Director

Table of contents

A2.1. Introduction
A2.2. Catalytic efforts
A2.3. Standards and best practices
A2.4. Pooled research
A2.5. Communications

A2.1. Introduction

From its inception in 1995, the DLF set itself the task of "creating the means to bring together digital materials - from across the nation and beyond - that will be made accessible to students, scholars, citizens everywhere". As it enters its sixth year, it is settling on a process for achieving this ambitious aim. In particular, the DLF is demonstrating its ability in:

  • incubating new services and tools that digital libraries commonly require but cannot independently develop or sustain;
  • identifying, endorsing, and promoting the standards and good and best practices that support the digital library's interests in high-quality, interoperable, and sustainable online collections and services;
  • pooling members' research and development effort for common benefit; and
  • facilitating communication amongst DLF members and ensuring that knowledge gained by the DLF is transferred into the broader community.

This document lists initiatives that the DLF has sponsored since its inception and includes initiatives that are planned but not yet formally begun. [Cost indications have been omitted from the public draft].

Initiatives are grouped with reference to the DLF's aims as:

  • Catalytic (work incubating services etc)
  • Work on standards and best practice
  • Pooled research
  • Communications

A2.2. Catalytic efforts

DLF funding stimulates the development of so-called infrastructural or utility tools and services that are commonly required by digital libraries but beyond their independent means. Work in this area develops functional specifications and business cases for the development of these tools or services. Working in this way, the DLF seeks to reduce the risk to investors who have the financial capacity to develop such tools and services. In this respect, DLF investment may be seen as incentive or venture funding.

Project Description and impact
Academic Image Cooperative

January 1999 - August 2000

The DLF's work prototyping organizational, business, and technical aspects of an image distribution service contributed directly to the formation and launch recently by the Mellon Foundation of ArtSTOR - "an independent not-for-profit organization that will develop, 'store', and distribute electronically digital images and related scholarly materials for the study of art, architecture, and other fields in the humanities". ArtSTOR marks a major advance in the development and dissemination of visual image resources that support research and teaching. [Text deleted]

The DLF continues to work closely with ArtSTOR.

Shared cataloguing tool for visual resources.

January - April 2001

ArtStor has also subsumed the DLF's work on a tool to reduce redundant effort involved in cataloging visual resource. Beginning in January 2001 the DLF developed an outline functional specification and market assessment for the cataloguing tool
Open Archives Initiative (OAI).

March 2000 - October 2002

Through its two-year investment in the OAI, the DLF has helped to develop a network protocol upon which the next generation of scholarly Internet portal services is likely to be built. With CNI and the NSF, the DLF helps to support:
  • the OAI's steering committee (its governing body - the DLF director is its chair);
  • the OAI Executive (located at Cornell University)
  • the OAI technical committee

The DLF has also facilitated the development of a number of Internet gateway services, funding for which is being supplied by the Andrew W. Mellon Foundation [text deleted].

The OAI is having significant impact in the US and abroad. It is being evaluated by RLG and OCLC as a means of exposing bibliographic records to the world wide web (something that will fundamentally change bibliographic provision and library service); it is a building block of the NSF's National Science Digital Library; it is being evaluated in the UK as a means of integrating access to higher education information services; it is widely adopted by scholarly preprint repositories in the US and abroad; it is being considered by some publishers as a means of revealing. In 2002, the OAI will be developing an exit strategy and the DLF will not expect to support its activities after October of that year.

Minimum requirements of digital archival repositories for electronic scholarly journals.

January 2000 -

In early 2000, CLIR, the DLF, and CNI initiated a process through which they developed consensus amongst publishers and libraries around minimum requirements of an e-journals digital archival repository. That consensus helped encourage The Andrew W. Mellon Foundation to launch its e-journals archiving program and fund seven institutions [text deleted] to plan the development of such repositories. In a second phase of the program to be launched in 2002, it is hoped that some of the planning projects will be funded to move to an implementation phase.

With this program, libraries and publishers take a significant and practical step forward in addressing their shared preservation concerns and developing repositories for e-journals. The DLF remains closely involved in the Mellon e-journals archiving program, hosting its web pages and facilitating discussion and information sharing amongst its participants.

Localized reference linking.

February 1999 -

With NISO and other organizations, the DLF has developed a reference linking architecture that will enable a university library to resolve reference links to text documents to which they have access. By endorsing this "localized linking" architecture, and promoting it to other library associations (e.g. ARL, ICOLC, etc.), the DLF was also able to encourage its evaluation (and, we hope, ultimate )adoption by CrossRef, an organization of some 30+ publishers that is implementing a reference linking service involving its members' numerous journal publications.

The adoption of this localized linking architecture will benefit libraries economically by allowing them to benefit from reference linking systems without constraining their choice of supplier for citation and full-text content.

Registry for digitized books and serials

April 2001 -

An increasing number of libraries and commercial entities are involved in converting existing paper-based books and serials to digital form. Unlike the special collections materials that have been the focus of digital conversion in many libraries, books and serials are commonly duplicated in many different institutions. This presents both an opportunity and a threat. The opportunity is for coordination between institutions, with the efforts of each contributing to a larger shared but distributed collection. The threat is that resources will be wasted in the repeated digitization of the same material.

A key requirement to realizing the opportunity and avoiding the threat is a mechanism for sharing information in a coherent fashion between institutions about what has been digitized; that is, the creation of a Registry of digitized materials.

The DLF has developed a functional specification and business case for this registry and is currently in discussion with potential service providers who are positioned to supply it.

The purpose of the Registry is to provide a place for institutions that have created (or are otherwise responsible for) digitized versions of traditional printed books and serials to record:

  • what specific items have been (or are about to be) digitized;
  • where they can be accessed;
  • the specifications followed in digitization.

Such a registry promises numerous benefits to libraries, including the reduction of duplicative effort and in enhanced access to digitized books and serials.

Tool for the construction of benchmark quality Encoded Archival Descriptions (EADs).

Kick-off meeting is scheduled for autumn 2001

As more archival finding aids become available for online exploration, scholarly and curatorial communities anticipate being able to search seamlessly and easily across them. Perhaps the single greatest obstacle to realizing this vision, is the enormous variation in the way that archival finding aids are defined and created. This variation persists even as the Encoded Archival Description (EAD) emerges as the preferred encoding scheme for archival finding aids. Quite simply, there is insufficient agreement about how beset to implement the EAD to create genuinely interoperable online finding.

With this tool, the DLF hopes to encourage adherence to some minimum-level benchmark requirements and thus to support the broader aim of developing interoperable finding aids.

The DLF will define the functional requirement of this tool and facilitate its development by an appropriately positioned service provider. The work builds directly on the DLF's successes defining a functional requirement and business case for a visual resources cataloguing tool, and in finding a service provider to develop that tool as a production service.

Shared digitization service

Kick off meeting scheduled for autumn 2001

The DLF is developing the functional specification and business case for a service that reliably produces digital surrogates for items in general and special collections. It will then facilitate the development of such a service by some appropriately positioned third party.

Such a "shared digitization service" would allow libraries large and small to undertake digitization programs without having to rely so heavily on (a) commercial entities that may not well understand library needs or (b) in-house production units that are typically costly, small in scale, and financially insecure.

Repository for heterogeneous digital objects

Kick off meeting scheduled for October 2001

The DLF is supporting an initiative in which several institutions will develop prototype digital repositories using a common repository architecture that has been developed to coherently and consistently manage large-scale collections of deeply heterogeneous digital objects (e.g. digital images, digital texts, digital sound recordings). DLF support for the initiative runs to an initial meeting of its participants and is intended in part to assist the initiative in attracting substantial external funding as required for full implementation.

The meeting is planned for October 5-6, 2001. I is premature to assess its outputs and impacts.

Repository and alerting service for open source digital library software.

Kick off meeting scheduled for October 2001

The DLF is supporting a meeting to investigate what steps it might take to ensure that its members (and digital libraries more generally) gain more advantage from increased development in libraries and library systems suppliers of open source software and tools.

The meeting is planned for October 5-6, 2001 so it is premature to assess its outputs and impacts.

A2.3. Standards and best practices

Leveraging its members' collective influence the DLF identifies, endorses, and promotes those standards and good practices that support the digital library's interests in high-quality, interoperable, persistent, and sustainable online collections and services. It is also developing the network of strategic alliances it will need to ensure that its good and best practice recommendations are maintained and communicated to the broadest possible community.

Although there is an opportunistic element in the way that priorities are assigned to work in this area, effort increasingly grows out of the best practice requirements that surface in developing functional and business cases for essential utility tools and services (see Appendix 2, Section A2.2.).

Project Description and impact
Liblicense model agreement

- June 2001

(The model license was endorsed by the DLF in January 2001).

This model license effectively documents preferred and good practice and acts as a decision tool that is likely to save libraries both time and money in negotiation robust contracts with commercial content providers. Ann Okerson (Yale University) will maintain the model license so that it evolves to reflect new circumstances as they arise.

The model license helps to introduce consistency and economy of effort in library licensing practices and has been taken up widely by both libraries and publishers in the US and abroad.

Structural, technical, and administrative metadata for digital objects (Making of America 2, and METS).

MOA 2 September 1996 - December 1999

METS Feb - Nov 2001

DLF has supported work on mechanisms for describing technical, structural, and administrative characteristics of digital objects. Initial recommendations about technical, structural, and administrative metadata emerged from the Making of America II initiative (see http://www.diglib.org/standards/dlfmoaii.htm).

In recent months, these recommendations have been refined and extended by a DLF working group into a more inclusive "Metadata Encoding and Transmission Scheme" (METS). Work of the initiative is fully documented on a website that is being maintained by the Library of Congress (http://www.loc.gov/mets/. The site includes an overview and tutorial, a beta version of the METS schema (in XML), an example METS XML document, and METS documentation.

A common means of recording attributes of digital objects is a fundamental building block of any distributed digital library service. It is as essential to institutions that wish to integrate access to or management of their own distributed digital collections. Widespread adoption of the METS standard as it evolves will therefore have a significant impact.

Benchmarks for digitally reformatting printed books and serials

June 2001 -

Libraries and others are digitizing increasing quantities of printed material for online access without agreement on any desirable level of imaging quality. The DLF is working to identify, and build support for, specifications acceptable as the minimum necessary for digitally reproducing printed books and serial publications with fidelity.

Adoption of such benchmarks would help users and libraries both. Users could have more confidence in the fidelity of digital reproductions available to them. And libraries could produce and maintain reproductions with confidence that expensive re-digitization would not become necessary. Digital reproductions meeting at least the benchmarks' minimum specifications would remain viable even as reproduction techniques improved. Also, because such texts would have well-known, consistent properties, they could support a wide variety of uses (including uses not possible with printed texts). Additionally, agreement on minimum benchmarks for digital reproductions of printed publications is an essential first step for libraries that wish to investigate whether they could manage and preserve print materials more effectively if they relied more heavily on digital reproductions for access.

Methods for evaluating digital image quality

March 2001 -

Significant investments are being made by cultural organizations in the digitization of the pictorial collections that exist in libraries, archives and museums. The activity is highly decentralized. In the absence of guidelines or best practices for image digitization, one concern is whether the resulting digital collections will be interoperable across curatorial domains: are images that are made in libraries, museums, and archives, being made in formats that are easily distributed; to a baseline level of quality that meet users' needs, etc.?

In March 2001, the DLF sponsored a meeting in Boston to establish a forum for expert practitioners to exchange ideas about what is "good" and if possible to prioritize where tools, applications, and training will be of greatest benefit to institutions interested in making digital reproductions of consistent quality and persistent utility.

The forum continues and is pursuing a number of practical investigations of its own design (e.g. into the impacts of digital libraries of JPEG 2000). It has also been invaluable to the DLF as a source of expertise on other benchmarking issues that arise in consideration of digital image content.

Implementation guidelines for the Visual Resources Association Core Categories version 3.0

September 2001 - September 2003

The Visual Resources Association has developed an outline scheme for describing works of art. What the scheme lacks is implementation guidelines that will encourage visual resource cataloguers to develop visual resource descriptions that conform at least along minimal lines.

The absence of such guidelines emerged as a critical obstacle to the development of a shared cataloguing tool for visual resources and even in the development and deployment of an image distribution service such as ArtSTOR.

DLF support for the development of these guidelines has helped to attract matching support from the Getty Research Institute.

Benchmarks for digital archival repositories for electronic journals

January - September 2000

In early 2000, CLIR, CNI, and the DLF initiated a process through which they developed consensus amongst publishers and libraries around minimum requirements of an e-journals digital archival repository. That consensus helped encourage The Andrew W. Mellon Foundation to launch its e-journals archiving program in which seven institutions [text deleted] are planning the development of such repositories.
Benchmarking digital reference services

August 2001 - August 2002

As the number of digital (online) reference services grow in library and commercial communities, so does the need for some means of assessing their value, quality, and performance. As library patrons demand more services online, and as reference librarians seek to better meet patrons' information needs through the Internet, it has become essential to determine common definitions of success and quality. Library administrators need strong, grounded metrics and commonly understood data to support digital reference services, assess the success of these services, determine resource allocation to services, and determine a means for constant improvement of digital reference within their institutions.

For these reasons, the DLF is working together with OCLC, the Syracuse University, Florida State University, the Library of Congress and others to develop methods to assess the quality of digital reference services, test and refine measures and quality standards to describe digital reference services, and to produce a guidebook that describes how to collect and report data for these measures and standards.

TEI text encoding in libraries.

June 1998 - July 1999 (guidelines endorsed by the DLF in January 2001)

In the early 1990s, the Text Encoding Initiative prepared guidelines for the use of SGML for encoding texts. They did not, however, propose implementation guidelines. Accordingly, as the number of TEI-encoded texts grew, so did the variations in the application of the TEI encoding scheme. Ironically, the absence of implementation guidelines or minimum requirements impeded the TEI from achieving its key aim: encouraging the interoperability and exchange of encoded texts.

To address this situation, the DLF supported the development of guidelines pertaining to the application of the TEI and particularly "best practices" for the encoding of electronic texts developed for different purposes.

The guidelines have been endorsed by the DLF and are in use by leading text centers in the US and Europe.

A2.4. Pooled research

The DLF pools research and development expertise at member and other institutions to survey and evaluate existing practice and to answer pressing strategic, organizational, and technical questions. Work in this area is driven by members' perceived needs, by opportunities that are available to conduct meaningful research, and, increasingly, by the questions arising from DLF investigations into infrastructural tools and services (see Appendix 2, Section A2.2)

Project Description and impact
Strategies for developing sustainable digital collections

April 2000 - July 2001

The development of useful and sustainable digital collections is one of the greatest single challenge confronting digital libraries. Whether the digital information comes from a commercial publisher or from a digitization unit within the library, it seems to exist under a cloud of profound and unsettling uncertainty. Will it be useful and useable in its present or intended form, or require additional work on the part of catalogers, systems staff, or subject bibliographers? What new demands will its availability make on library reference staff? What level of continued investment will be necessary to ensure its accessibility on current hardware and software?

In April 2000, the DLF commissioned three reports to address these questions basing their analysis on the substantial experience that leading research libraries are amassing with their digital collections. The reports deal respectively with commercial electronic content, digital materials created from library holdings, and Web-based "gateways" that link to selected Internet resources in the public domain.

Adhering to a common outline, the reports demonstrate how decisions taken by a library when acquiring (or creating) electronic information influence how, at what cost, and by whom the information will be used, maintained, and supported. By assembling and reviewing current practice, the reports aim where possible to document effective practices. In most cases, they are able at least to articulate the strategic questions that libraries will want to address when planning their digital collections.

The reports are only now being published so it is premature to judge their impact. Certainly, response to public presentations based on earlier drafts has been very promising.

Digital library policies, organizations, and practices.

January 2001 - January 2001

The DLF has completed a member survey designed to identify the institutional contexts in which digital libraries are being developed. By documenting the very different paths along which digital libraries evolve, and the different ways they organize and fund themselves, the survey will: inform strategic planning and decision making within digital libraries; provide benchmarks for assessing digital library development; and identify emerging library roles.

A draft report based on the survey data is available from the DLF website. A formal report that will contain case studies that document the key digital library development paths that are apparent in the data will be available at the end of the calendar year. Accordingly, it is premature to assess the initiative's impact.

The library's role in the broader scholarly information landscape. In March 2001 the DLF convened directors from eight academic libraries to consider a white paper on how and why library use is changing, and to discuss how DLF could enhance and extend efforts to assess and then understand the likely impacts of those changes. Given the emphasis by ARL and others on developing methods for measuring the provision and impacts of library collections and services, it was agreed that the DLF might focus on broader scholarly information landscape to which those collections and services contributed one part.

In response, the DLF is working with Outsell Inc. on a study that will reveal the extent, nature, and use of that scholarly information landscape at leading liberal arts colleges and research universities. This knowledge will be invaluable for libraries and universities in planning information services to focus explicitly on the current and emerging needs of their faculty and students, and to avoid focusing on what is not, or may no longer be, important. The academic community will also benefit as publishers and content providers that serve the education market create better information products based on an increased knowledge of users' needs.

When the study is completed, the DLF will follow up with some of the libraries whose directors were initially convened to shape the study. Working closely with those libraries, it will examine how the study's results help to interpret local library trends (for which data exist, for example, through ARL and other bodies) and how survey results might or should impact upon library planning.

DLF has commissioned the study design from Outsell Inc. and is seeking external funding to support implementation. Both the study design and a funding decision are expected on 9/15/01.

Methods for assessing use of online collections and services

January - October 2001

How to assess the use and usability of online library collections and services is a significant challenge that confronts DLF members and other leading digital libraries. Such assessment is critical for evaluating the impact of online activities. It is as crucial for library planning.

Though nowhere formally documented, leading research libraries have a wealth of relevant assessment experience. Surfacing that experience is the aim of a survey that is being conducted by DLF Distinguished Fellow Denise Troll (Carnegie Mellon University). Ms Troll's survey looks at assessment practices at DLF member institutions and is based upon 65 interviews. It summarizes, documents, and assesses current practices, highlights lessons learns and focuses on practices that have proven to be particularly effective. The survey report will include illustrative case studies. Although it is not intended as an tutorial in assessment methods, it will supply methodological overviews and provide extensive pointers to those who need to know more.

A draft of the study has been completed. Final publication is expected in Fall 2001.

Issues in electronic scholarly publishing

March 2001

In May 2001, the DLF convened a small group of representatives from university libraries and university presses in order to:
  • discuss their different approaches to electronic scholarly publishing,
  • establish a venue for information sharing about appropriate tools and technologies, and
  • encourage collaborative developments.

The meeting was intended as a first step in an exploration of production and distribution needs that may be common to libraries and university presses and as such met through some common solutions.

To date, no concrete results have materialized from the meeting.

How humanities scholars use and value library information resources. A study commissioned by the DLF into how humanities scholars locate, explore, and use research materials in print and digital formats.

The report (to be published in Fall 2001) is likely to have profound implications for how libraries prioritize investment in online collections and services.

Although it is too soon to assess the study's impact, preliminary findings were presented to the DLF Fall Forum 2000 to significant acclaim.

Outsell research and information service . The DLF subscribes to this information service.

Weekly reports are circulated to DLF members. Value of these reports has yet to be assessed.

Digital certificates Under DLF auspices, the California Digital Library, Columbia University, JSTOR, and OCLC developed a protocol that enables an information resource provider to verify that a user bearing a digital certificate has authority from a home institution to use a requested resource. The prototype system combines the use of X.509 digital certificates for authentication with a directory service providing authorization to licensed resources based on user attributes.

The work produced an architecture statement and contextualizing material and has been very influential in shaping direction of research and practical experimentation with certificate based authentication systems including work undertaken by Intenet2.

Working with distributed finding aids. Report commissioned by the DLF on the means and costs of searching encoded finding aids that are distributed at different institutions.
Workshop on social science and government data libraries. In January 1999 the DLF supported a workshop hosted at Princeton University to explore common needs as felt by social science data managers and other experts into: the discovery and retrieval of databases, the evaluation and interpretation of alternative data sources, and data extraction for analysis and presentation.

Workshop participants identified a number of concrete next steps in a report that provides a useful survey of challenges and opportunities in the social science data community.

No follow up work has been conducted.

A2.5. Communications

To build a community of professionals, the DLF facilitates its members' communication, collaboration, and cross-fertilization. Early indications from an evaluation instrument recently circulated to staff at member institutions suggest the DLF's newsletter (now published twice rather than four times a year) and its extensive and growing website are well received.

Communications component Description and summary of survey response
DLF mailing lists. The DLF uses an "announce list" to communicate with staff at DLF member institutions about its initiatives, publications, etc.

The list, with 96 members (126 when DLF directors are included) is well regarded by members who have asked us to advertise its existence (and methods for joining) more widely. We are also encouraged not to use the list to circulate information and announcements originating outside the DLF.

The DLF Web site. The Web site was comprehensively redesigned and reorganized in Spring 2000 to anticipate growth in the number of DLF initiatives and accommodate new reporting mechanisms (e.g. the Newsletter, the registries, the regular reports from the Director's Office).

The Web site is well consulted and generally found useful and navigable. Nonetheless, respondents emphasized the need for the DLF to alert people to new additions to the site as they are posted. And respondents made several specific suggestions for improvement of the site that will be taken up over the next several months.

The DLF Newsletter. The DLF Newsletter includes regular reports from member institutions on their digital library developments.

It is well received and will continue to be published twice a year (requiring annual reports from DLF member institutions) and members will be notified of its "publication".

DLF printed publications. These are the edited and highly polished reports that result from DLF-sponsored initiatives. The reports have been integrated into CLIR's "burgundy series" lending them credibility and, we hope, strengthening the series as a whole.

A "communications survey" commissioned by CLIR found that the publications are very well received and regarded.

DLF Directors' reports. These regular progress reports appear in the Newsletter and summarize progress of DLF initiatives.

The reports are generally well received. In future, attention will be paid to developing bulleted summaries that emphasize the significance of the DLF's various projects, and notifying members of new reports as they are posted.

DLF project reports. These more detailed reports chart work in progress and summarize initiatives with results that do not lend themselves to publication in the "burgundy series."

Project reports are particularly well appreciated by project participants (they help to track and focus effort). They are also well received by other specialists who find their way to the reports out of interest or need. They are not often read by people who come to the DLF website with a more general set of interests. The generalists who do read these reports find them to be too detailed or too technical. The DLF will continue to publish these reports on its Web site and will take steps to ensure that they attract and are readily accessible to the appropriate, specialist audience for which they are intended.

DLF registries. The DLF has three registries: a calendar of digital library events sponsored by the DLF or member institutions; a web-searchable database of policies, strategies, working papers, standards and other application guidelines, and technical documentation developed by DLF members to inform or reflect upon their digital library development; and a web-searchable database of members' public-domain, online digital collections. A majority of people who responded to the communications survey do not consult the registries, and of those who do, few find them particularly useful. In light of these comments it is proposed to discontinue the events calendar and seek means of promoting and encouraging use of the digital collections registry and the documents database.
The DLF Forum. The Forum is hosted periodically to enrich, stimulate, and inform the work of staff at DLF member institutions and to showcase and gain input into current DLF initiatives.

A survey conducted after the Fall 2000 Forum fall suggests that the forum is a great success and fills an important void for information professionals actively engaged in the development of digital libraries. The survey also suggests that the forums should: continue for the present on a twice-yearly basis; take place over two full days; provide ample opportunity to discuss and develop current DLF initiatives; provide opportunities to hear from representatives of funding agencies and other bodies that exercise some strategic influence over or leadership in digital library development.

Appendix 3. Digital Library Federation Review. Survey of members and other decision makers. Executive Summary

September 3, 2001

Conducted by: The Communications Office, Inc.
108 East Del Ray Avenue
Alexandria, Virginia 22301
(703) 567-4977

Revised by the DLF Review Panel to Protect Confidentiality

Table of Contents

A3.1. Survey of DLF Steering Committee: Findings and Highlights
A3.2. Survey of Key Informants: Findings and Highlights

A3.1. Survey of DLF Steering Committee: Findings and Highlights

  1. Why are you a member of the DLF?
    The theme for many was "collaboration." There appears to be an overarching philosophy that the sum of the work of the DLF is greater than its parts. Members are eager, especially for their staff members, to share information and problem solving. With the tremendous volume of work to be done, there needs to be economy of effort among members. The majority of respondents see their institutions as contributing strong currency to the collaborative process that will benefit the "whole." A few talk about what DLF can provide to their individual institutions. It is important to members that the DLF's work fill a unique niche and not be redundant in areas addressed by other organizations. Many Steering Committee members talked about the DLF's value to their staffs' development.
  2. Has the DLF enabled its members to accomplish what could not have been done individually or informally?
    The unanimous response to this question is "yes."
  3. What has the DLF enabled its members to accomplish?
    Once again, the opportunity to collaborate was a central theme. More specifically, the Open Archives Initiative metadata harvesting study and facilitated access to grant money were often mentioned. Among the other responses were cross reference mobilization, survey research, identification of best practices and models, metadata standards, preservation of electronic journals, registry of digitized books and serials, the academic image cooperative (ArtSTOR), and standards work such as METS.
  4. Has your institution profited from its investment of time, money, and attention in the DLF? What value has it received?
    Every institutional member said yes, they have profited from their investment in the DLF. Again, they noted the value of collaboration and facilitated access to grant money. There is no doubt that the DLF forums are highly valued. At this point in the survey, a few comments emerged indicating that some Steering Committee members believe the collaborative value of DLF rests more with university staff members who are directly involved in digital programs, not with the Steering Committee members. This theme recurs more strongly further into the survey.
  5. How can the DLF better assist you in leveraging your investment?
    The spectrum of responses was broad, as indicated in the following sample responses. Once again, there were comments urging more inclusion, in terms of library staff from DLF institutions. Members want DLF to take a stronger, more visible leadership role on digital issues. The belief is that with strength will come more funding opportunities for the organization and its members. They would like to see their investment in the DLF reap more grant dollars.
  6. Has your rationale for belonging to the DLF changed over the years? If so, how?
    About half of the respondents said their rationale for belonging to the DLF has changed. Many said their expectations for DLF have changed positively as the organization has matured. Once again, the subject of giving library staff a more active role in the DLF was discussed by a few. There appears to have been a shift in mindset as members initially set out to create one digital library; whereas today, they are working on a collaborative model built on multiple institutional contributions. There is concern expressed by a few that, as DLF has grown, its power and energy have been diluted by members who do not have the passion and expertise upon which the organization thrived initially.
  7. What has the DLF accomplished as an organization?
    Many people listed specific accomplishments, such as a registry for digital objects, reference linking, standards for digitization, archiving, best practices, and work user studies, among others. Providing opportunities for collaboration, bringing visibility to digital libraries and their issues, and attracting funders were more broadly defined accomplishments. Yet, others felt accomplishments were still pending and that more emphasis should be placed on outcomes.
  8. In what areas, if any, has the DLF developed a leadership role?
    Members view leadership from many perspectives. Some were quick to point out specific initiatives-- digitization, open archives initiative, metadata standards and metadata harvesting, and most notably, best practices. Others judged DLF's leadership status by prominence in government and funding agency meetings. Still others believe its leadership is best characterized by its catalytic ability to inspire collaboration and novel thinking. One member took exception to the term "leadership," preferring to cite "success" in providing positive outcomes to benefit scholars.
  9. Should the DLF continue?
    The answer was a resounding "yes." One member said "maybe," adding "I think it needs regular reviews. If it is not accomplishing stuff, it should go out of business."
  10. Should the DLF operate at its current level of funding?
    There was no consensus. Some members expressed a desire to better understand DLF's future direction before they could provide a definitive answer. Others approached the question from their institution's ability to pay higher dues, as opposed to assessing the best interest of the DLF. Of that group, some said they would not be willing to pay more; a few said they would be happy to pay more if the increase reaps greater benefits; still others thought an increase might be warranted, but it should be absorbed through additional members. Also, it is interesting to note that some equated an increase in funding with a larger organization, as opposed to expanded project funding.
  11. What funding sources should support the DLF?
    Members-to varying degrees-believe the current membership dues structure, combined with foundation and government grants, is appropriate. Other suggestions included commercial sponsorships and creating an entrepreneurial consulting venture for DLF.
  12. Should the DLF remain a limited-term organization, or should it establish itself as a permanent organization?
    Most of the respondents believe DLF should remain a limited-term organization.
  13. What is the optimal size for the DLF?
    There was no consensus on optimal size. In distilling the content of the various responses, it would be accurate to say most members believe the current size is manageable (although a few feel that 26 is too large). However, there are those who feel the DLF should be more inclusive and open membership to an unlimited number of institutions that are able to meet the organization's high standards and dues requirements. These particular respondents are not necessarily advocating for a large DLF membership; they believe that the high entrance standards and financial commitment will effectively regulate the size of the membership.

    Others were more concerned about the quality than the actual number of members. This topic was introduced earlier in the survey.

    This question elicited talk about the potential of changing organizational structure to accommodate the present membership and, definitely, a larger one. That topic is addressed in greater detail further in the survey.

  14. To whom should the DLF expand its membership?
    [Text deleted]
  15. What institutions, if any, should DLF expand its membership?
    [Text deleted]
  16. Is the DLF's current organizational and governing structure effective in meeting the work and leadership requirements of the organization?
    About half of the respondents believe DLF's current organizational and governing structure is effective. However, many of them provided a qualified "yes." The creation of an Executive Committee to make quick decisions and keep the organization nimble has been welcomed. There are concerns about the function of the Steering Committee. Some say it is ineffective because of its size; others say it is ineffective because its members are only peripherally involved in the day-to-day, hands-on operation of digital library programs. The theme of engaging more project people in the DLF emerges again.
  17. What improvements, if any, might you suggest in DLF's current organizational and governing structure?
    [Text deleted.]

    The Steering Committee continues to be a focus of concern for some, in terms of its personality and effectiveness. There is a feeling that some of the library directors do not have the expertise or interest to stay actively engaged. This points back to the contention by some that the Steering Committee members should address DLF administrative and broad programmatic issues; those with specific digital library expertise should engage one another and spearhead collaborative efforts.

  18. What has been the most successful part of the DLF agenda?
    As one can see from the following comments, members see a number of DLF successes. The overwhelming response is the DLF forums, followed by the Open Archives Initiative. It is obvious, reviewing these responses, that members find opportunities to engage their staffs in the work of DLF to be highly desirous.
  19. What has been the least successful part of the DLF agenda?
    Once again, issues of communication and Steering Committee dynamics were discussed. Other topics were mentioned-limited publication dissemination, lack of strategic planning, disconnects between endorsements and implementation, borne-digital vs. retrospective priorities, funding brokering, etc. However, relative to the comments provided in the rest of the survey, this "least successful" question yielded a limited number of answers.
  20. What might the future emphasis of the DLF be?
    Members have such a diverse "wish list" for DLF that it is not possible to identify particular trends. The responses speak for themselves.
  21. Does the DLF need other collaborators to achieve its goals?
    Most respondents see the necessity for collaborators, but they are quick to point out that the DLF membership should remain as it is. They do not see collaborators as future DLF members. A few members are willing to explore collaborative relationships with commercial vendors, but a general distrust of that group was expressed. Yet, others express a desire to at least stay abreast of commercial developments that could benefit the DLF.

    [Text deleted.]

A3.2. Survey of Key Informants: Findings and Highlights

  1. Have DLF initiatives advanced the work of the digital library community?
    All respondents answered "yes."
  2. How has the DLF advanced the digital library community's work?
    Most respondents answered in the broad sense of providing a forum for collaboration, setting standards, conceptualizing and articulating necessary agendas, promoting staff development, infrastructure development, etc.

    [Text deleted.]

  3. Has any DLF work influenced how you've developed your digital library program?
    This question was not relevant to all respondents. University librarians cited influence in the area of standards, infrastructure issues, and shared developments and approaches. [Text deleted.]
  4. Has the DLF enabled 'the community' to accomplish what it could not do individually or informally?
    The consensus was "yes," but with qualifying statements:

    [Text deleted.]

  5. Should the DLF continue?
    Of the eight respondents, only five gave an unqualified "yes."

    [Text deleted.]

Who is The Communications Office?

Digital Information issues are not new to The Communications Office. It recently conducted a similar survey for the Council on Library and Information Resources (CLIR). The firm counts among its clients the Library of Congress, Congressional Research, and National Archives & Records Administration. It wrote the Digital Library Federation's organizational brochure ("Chaos and Revolution") under the direction of Don Waters as well as the collateral materials for CLIR's "Into The Future" digital memory crisis campaign in 1998.

return to top >>