Evaluation of the Digital Library Federation, 1995-2001.
Summary Report of the review
panel as endorsed by the DLF Steering Committee
Submitted on September 28,2001 by the DLF review panel:
Bernard Hurley, University of California at Berkeley
Paul Mosher, University of Pennsylvania
Martin Runkle, University of Chicago
Gloriana St. Clair, Carnegie Mellon University
Sarah Thomas (Chair), Cornell University
As amended and unanimously approved by members of the DLF
Steering Committee on November 14, 2001
In June 2001 the Digital Library Federation (DLF) Steering
Committee approved the creation of a Review Panel to evaluate the
progress of the DLF in achieving its goals and to consider the
What follows is the Panel's report as endorsed by the DLF
Steering Committee at its meeting on November 14, 2001. The
report explains why and how the evaluation was conducted and
outlines its principle recommendations. It also supplies evidence
submitted to the Review Panel in the course of its work.
Table of contents
General Findings and Recommendations
Appendix 1. Review of the Digital Library
Federation's progress meeting its aims, goals, and objectives
Appendix 2. DLF initiatives: description,
impacts, and costs
Appendix 3. Digital Library Federation
Review. Survey of members and other decision makers. Executive
In June 2001 the Digital Library Federation (DLF) Steering
Committee approved the creation of a Review Panel to evaluate the
progress of the DLF in achieving its goals and to consider DLF's
future. The founders of the National Digital Library Federation,
DLF's precursor, had placed an initial five-year limit on the
organization, and in its strategic plan approved in March 2000,
the DLF Steering Committee had recommended that the DLF conduct
an assessment before the conclusion of the DLF funding cycle in
June 2002. The DLF Executive Committee formed a Review Panel to
meet the charge of the Steering Committee.
The charge noted that the DLF evaluation should:
- Be undertaken by a Review Panel consisting of members of the
DLF Steering Committee
- Assess progress to date and help shape any future program
- Consider whether, how, and under what organizational and
financial terms the organization might be continued; and
- Result in a report from the Review Panel to the Steering
Committee by 1 October 2001.
The Review Panel identified key data and information it
required as background to evaluate the DLF. At the Review Panel's
request, the DLF Director, Dan Greenstein, prepared a review of
DLF's progress meeting its aims, goals, and objectives; a
description of DLF initiatives, their impacts, and the costs
associated with them; and other documentation relating to DLF
activity (Appendices 1 and 2). DLF engaged consultants to survey
members and a select group of non-members using a list of
questions prepared by the Review Panel (
Appendix 3). The Review Panel also
interviewed the DLF Executive Director and the Council on Library
and Information Resources (CLIR) President. The Chair of the
Executive Committee was also consulted. The DLF Review Panel has
created its report drawing on these rich and varied inputs.
General Findings and
DLF members unanimously agreed that the Digital Library
Federation has had a significant, positive impact on digital
library development and that the DLF should continue its work.
There was also strong consensus that the DLF benefits from
Recommendation 1: The Digital Library Federation
should continue for a limited term of five years (July 2002- June
2007) and should undergo a subsequent review in 2006.
To guide the DLF over the next five years, members called for
the creation of a new strategic plan that would establish
directions and priorities for its activities commensurate with
Recommendation 2: The Executive Committee should lead
the Steering Committee through the development of a strategic
While the DLF Member Survey revealed strong support for the
current direction, productivity and leadership of DLF, it also
reflected some confusion about the governance structure of DLF
and its relationship to CLIR. Although informal structure was
appropriate when the organization was young, the membership
small, and the values and goals common among membership and
staff, DLF may have grown faster than its institutional memory,
and now is felt by the members to need a clear, open and stable
governance structure understood by all.
Members would like the relationship between DLF and CLIR to be
well-defined, have an open system for nominating and electing the
Executive Committee, and be clear about the different functions
of the Steering Committee and the Executive Committee, whose role
most favor clarifying and strengthening. Members made clear that
they take the term "federation" seriously: the Federation
consists of member institutions who contribute substantially in
funds and staff involvement. It is a participative, contributive
organization rather than a membership or service organization. It
is intended to be highly effective, have a smaller staff, a
leaner organization, and depend on members' expert staff and
third party agencies for program and project support rather than
We urge that the following recommendations be incorporated in
the DLF Bylaws, which should be drafted by the Executive
Committee and passed by the Steering Committee.
Recommendation 3: DLF is governed by its Steering
Committee, which consists of the senior executive of each
contributing member or its designated representative. The DLF
strategic plan, DLF program direction, membership criteria and
issues, and dues and fees, are set by the Steering Committee upon
the advice of the Executive Committee. The Steering Committee
shall meet at least twice annually, as scheduled by the Executive
Committee, to hear reports and transact the business of the
Recommendation 4: The Council on Library and
Information Resources serves as the administrative home of DLF.
The DLF Steering Committee exercises program and broad budgetary
direction for DLF; CLIR has fiscal responsibility for DLF, but
DLF funds are not intermingled with those of CLIR.
Recommendation 5: The DLF Executive Committee has
authority for setting meeting agendas for DLF, and for meetings
of the Steering Committee; it is responsible to the Steering
Committee for DLF governance between meetings of the Steering
Committee, and regularly meets between Steering Committee
meetings. The Executive Committee should act as a bridge between
the Steering Committee, the DLF Executive Director, and CLIR, as
Recommendation 6: The Chair and members of the
Executive Committee are elected by the Steering Committee from a
slate of candidates nominated by a committee of members appointed
by the Chair of the Executive Committee. There will be an
opportunity for nominations from the floor at the time the slate
of candidates is presented for election to the Steering
Recommendation 7: The DLF Executive Director is
responsible to the DLF Steering Committee through its Executive
Committee for the effective functioning of the Federation, and
reports periodically to the Steering Committee on the condition
and achievements of the Federation and its budget The DLF
Executive Committee exercises programmatic oversight over the DLF
Executive Director, while CLIR exercises administrative and
fiscal oversight. The Executive Director also attends meetings of
the CLIR Board, and reports periodically on the progress and
accomplishments of DLF.
Issue: Organizational Effectiveness
The Steering Committee survey found DLF and its leaders
uncommonly effective for a small, low-overhead organization. All
members agreed they had profited from their investment in DLF. It
is clear that DLF staff understand the goals and outcomes desired
by members, and that DLF has enabled members to accomplish what
could not have been done individually. DLF is fundamentally
understood as a collaboratory focusing on digital library issues
and projects of common concern. Frequently-cited examples of
success in this role are the Open Archives Initiative (OAI)
metadata harvesting study, access to grant fund support for key
projects, identification of best practices and models, work on
metadata standards, outcomes research, digital preservation, the
Forums, and digital preservation.
There was much discussion of organizational size in the survey
results. Consensus was that a size of 26-36 members is about
right, and that the criteria for membership should emphasize
participation and contribution. Genuine collaboration and
participation are what set DLF apart as an organization--shared
expertise was often cited as a major factor in DLF achievement.
Useful outcomes rather than cachet should remain DLF's goal.
Recommendation 8: Steering Committee meetings should
be thematic or emphasize issues of strategy, project direction,
or policy. Reporting should be handled when appropriate by
succinct written reports appended to the meeting agendas.
The Review Panel concluded that there were opportunities to
strengthen communication between the Executive Director and the
Executive Committee as well as with the Steering Committee. The
DLF Executive Director should discuss proposed DLF initiatives
with the Executive Committee, and calls for expressions of
interest in participation in program activities. This procedure
addresses concerns expressed during the survey that DLF members
did not always learn of projects until they were underway or did
not have a chance to join in them. DLF members, even with the aid
of the DLF web page and regular emails, had difficulty in
enumerating many DLF activities, suggesting that other forms of
communication beyond the home page and emails may be
Recommendation 9: The DLF Executive Director and
Executive Committee shall ensure that all DLF members learn of
new initiatives through an open call for participation to DLF
members, but this does not require appointing all who ask to
The DLF has achieved considerable visibility for its
accomplishments in its short lifetime. However, many non-DLF
members remain unaware of the benefits of DLF or its
contributions in advancing digital library developments. The DLF
needs to continue to build credibility and influence.
Recommendation 10: The DLF members and Executive
Director should make a concerted effort to publicize DLF
activities, best practices, and strategic vision through
presentations, publications, and other means.
The new set of publications initiated by DLF and CLIR responds
to a need expressed by many for more effective dissemination of
DLF outcomes and results.
Recommendation 11: The DLF may from time to time add
new members who have the capacity and commitment to contribute
substantial expertise to DLF's programs and initiatives and will
advance the goals of the DLF membership.
Issue: Adequate funding for DLF Initiatives
Recommendation 12: The DLF Steering Committee should
consider whether to continue to accept the CLIR contribution of
Recommendation 13: The DLF Steering Committee should
vote on increasing dues by $1000 to an annual dues of
Recommendation 14: The DLF Steering Committee should
approve and encourage some use of the capital fund to pay for
projects and initiatives that have previously been charged to the
Issue: Broaden Member Participation
The strength of the DLF is in its membership and all members
are expected to participate in its initiatives. The survey of DLF
Steering Committee commissioned by this Review Committee
identified three themes centered on member participation.
- The DLF Forums have been enormously successful.
Recommendation 15: The DLF Forums should
Recommendation 16: The Review Committee recommends
that member organizations which have not made presentations at
the Forums be encouraged to do so, as learning of their efforts
will benefit the entire Federation.
- Digital library project managers should have more
opportunities to participate in DLF activities
The desire to increase DLF participation for staff with
digital library management or development responsibilities was
the most cited request identified by the survey.
Recommendation 17: The Review Committee recommends
that the full Steering Committee discuss participation by digital
library project managers in DLF activities and establish means to
increase this level of participation.
- DLF working groups should have a broader member
The survey identified a perception, if not a reality, that the
same people are called upon to serve on working groups.
Recommendation 18: The Review Committee recommends
that the DLF administration, working with the Executive
Committee, broaden the pool of potential working group members.
This could be accomplished by making the formation of working
groups more visible, so qualified staff throughout the membership
can volunteer and be considered for inclusion.
Appendix 1. Review of
the Digital Library Federation's progress meeting its aims,
goals, and objectives
Submitted to the DLF Review Panel for its consideration by
A1.5. Directors' activity report for the period December 2000 -
September 2001 (text omitted)
A1.6 Summary of DLF website use, January 2000 - July 2001 (text
A1.7. Level and extent of participation in DLF
forums and working groups
A1.8. DLF publications and reports since
This document describes the aims, goals, and objectives of the
DLF and its accomplishments in achieving them. It is prepared at
the request of the DLF Review Panel and submitted to that Panel
for its consideration as it evaluates the DLF's progress to date
and its future possibilities.
The document makes explicit reference to the DLF Strategic
that was agreed by the Steering Committee in March 2000, and
assesses progress made by the DLF in meeting the goals and
objectives set out in that plan.
Section A1.2. lists initiatives that the DLF has sponsored
since its inception and includes initiatives that are planned but
not yet formally begun. For each initiative, the list supplies a
title, a brief description and assessment of impact where
possible. A statement of the initiative's cost to the DLF was
The strategic plan states that the DLF leverages its members'
reputations, investments, and research and development capacities
- Share and evaluate information about digital library tools,
methods, practices, trends, and strategies
- Stimulate and share in the conduct of necessary digital
library research and development
- Respond quickly and effectively to digital library challenges
as they arise
- Exercise some influence for the library community over a
rapidly changing information landscape
- Act as a catalyst in the development of innovative
information services and organizations, and as an agent of
learning for the profession
- Attract investment in essential digital library research and
- Build a community of professionals appropriate to the
development of digital libraries.
Progress in these areas is described below.
A1.2.1. Fostering shared evaluation of digital library tools,
. The DLF has done this by:
- Publishing surveys of digital library developments at DLF
members and other leading research libraries. Recent surveys have
focused on strategies for developing sustainable scaleable
digital collections; methods for assessing use of online
collections and services; and the origins, focus, organization,
funding, and policy environments surrounding leading digital
- Ensuring that DLF-supported initiatives begin by evaluating
libraries' needs and how they are currently being met at DLF
member and other leading research libraries
- Regularly convening DLF forums
- Promoting specialized discussions on DLF-hosted listservs
(e.g. on image quality, and on preservation of e-journals).
The impact of the DLF's work in this area may become evident
from survey components of the DLF review (
Appendix 1, Section A1.3.).
A1.2.2. Stimulating and sharing in the conduct of necessary
digital library research and development
The DLF does this by helping to define essential research and
development agenda, align partners as required to carry out those
agenda, and assist those partners in attracting the support
necessary to fulfill their objectives. Evidence of the DLF's
progress in this area is available here and in Appendix 1,
The DLF has:
- Defined a stable metadata harvesting protocol attracting
support for the protocol's maintenance, further refinement, and
application from the National Science Foundation, the Coalition
for Networked Information, The Andrew W. Mellon Foundation, and
the British Library - see acknowledgements on the Open Archives
- Defined and encouraged adoption of schemes for representing
technical, structural, and administrative metadata ("METS" is now
maintained by the Library of Congress - see
- Defined and encouraged commercial adoption (e.g. by CrossRef)
of localized reference-linking solutions
- Defined and encouraged development of and experimentation
with certificate based authentication systems - examples of their
use are available from Internet2
(http://middleware.internet2.edu/shibboleth/) and the Corporation
for Research and Educational Networking (CREN -
A1.2.3. Responds quickly and effectively to digital library
challenges as they arise
Through directors and staff at DLF member institutions, the DLF
stays abreast of pressing digital library issues. With its
lightweight organization, its nimble decision-making and funding
apparatuses, and its access to expertise in member institutions,
the DLF is able to respond quickly to issues that are
particularly pressing and to take advantage of opportunities as
they arise. Progress in this area and evidence of the DLF's
nimbleness is evident in work that has been undertaken quickly in
response to need and opportunity as it was identified, e.g. with
localized reference linking, certificate based authentication,
registries for digitally reformatted materials, etc.
A1.2.4. Influence over for the library community
The DLF seeks to exercise its influence over the broader
community by capitalizing on:
- Members' prestige and the profile of their visibly successful
digital library programs
- Its exclusiveness as a small and selective membership
- The timeliness and quality of its initiatives
- Its network into initiatives, funding agencies, and other
bodies that exercise some strategic influence over the
development of digital libraries and digital library
- The endorsement that the DLF gives to selected
recommendations and activities
The DLF's communications program (see
Appendix 2, Section A2.5) is essential to work
in this area.
Evidence of progress is available in:
- Programs of the The Andrew W. Mellon Foundation that have
built directly upon work conducted in DLF initiatives (e.g. the
e-journals archiving program, ArtSTOR - see
- and the Foundation's recent investment in metadata harvesting
- The guidance sought from bodies that have some strategic
influence over how digital libraries develop [text deleted]
- Attention to and adoption of practices and models that are
endorsed by the DLF (notably the Liblicense model license
agreement between libraries and publishers, recommendations for
the use of TEI guidelines in libraries, recommendations to
CrossRef pertaining to the use of localized reference linking
- Extent to which the DLF Director is asked to represent the
perspective of the DLF and its members (through public
presentations, visits, and participation on advisory panels,
etc.) and to bring these to bear on various digital library
initiatives and programs
- Growing interest in membership in the DLF. Since January
2000, 2 libraries (at the Universities of Washington and Illinois
at Urbana-Champaign) have joined the DLF and the National
Archives intends to resume its membership. 11 institutions have
expressed interest in joining and invited site visits [text
deleted]. Additional preliminary discussions have been had with
representatives of [text deleted].
- Growing use of the DLF website as evident in Appendix 1,
A1.2.5. Acting as a catalyst in the development of innovative
Here the DLF helps to define requirements and business cases for
such services, and to align partners and attract investment as
necessary to develop them. Evidence of the DLF's progress in this
area is available here and under Aim 2. The DLF has helped to:
- Define and encourage investment in a distribution service
(ArtSTOR - op cit) for digital images that support research,
teaching, and learning in the visual arts and other humanities
- Define and attract investment in a cataloguing tool (ArtSTOR
- op cit) that will support shared production of visual resource
descriptions that meet some minimum level quality criteria
- Define minimal requirements for and attracted investment in
planning e-journal archival repositories (op cit)
- Articulate the potential of and business cases for Internet
gateway services built with the metadata harvesting protocol and
attracted investment in such services from The Andrew W. Mellon
Foundation, The National Science Foundation, the European Union
Directorate General XIII, and the Joint Information Systems
Committee of the UK's Higher Educational Funding Councils)
- Define and seek investment in a registry service to record
information about digitally reformatted book and serial
A1.2.6. Attracting investment in digital library research and
The DLF has attracted funding for member institutions for the
Director's office. The DLF's support for work in various areas
has also encouraged more general investment flowing from a
variety of agencies into non-DLF member institutions. No attempt
will be made to assess progress in leveraging funding of this
DLF members leverage their membership in the DLF by attracting
external funding for various digital library activities. The DLF
Directors' Office has only a very partial knowledge of where and
to what extent members are successful in this regard. More
information may surface from survey components of the review. The
Director's Office knows, for example, that DLF efforts have
helped DLF members attract funding for:
- Work on e-journal archiving (from the Andrew W. Mellon
- The development of OAI harvesting services (from the Andrew
W. Mellon Foundation, the National Science Foundation)
- Practical experimentation with metadata repositories (from
the Andrew W. Mellon Foundation and the National Science
- Work on particular digitization and interoperability projects
(e.g. from the Institute for Museum and Library Services, the
National Endowment for the Humanities, The Andrew. W. Mellon
Foundation, and the Getty Research Institute).
External funding for DLF initiatives flows through its
administrative host, the Council on Library and Information
Resources. The DLF has attracted funding for:
- Work developing a prototype and defining a business case for
an image distribution service (from The Andrew W. Mellon
Foundation - $100,000 for work on the Academic Image
- Work on digital preservation (the DLF participates in a
contract between the Library of Congress and CLIR through which
CLIR will assist the Library of Congress in planning a national
digital preservation program for which the Library of Congress
will receive up to $100,000,000 from the US Congress. The DLF's
role in this contract will be to help build a conceptual
framework within which a national digital preservation strategy
may be developed, and to define and implement the research agenda
as appropriate for developing that framework. More information is
available in Deanna Marcum, "CLIR Partners with LC on National
Program for Digital Preservation". CLIR Issues,
A1.2.7. Building a community of professionals
Here the DLF has attempted to encourage a culture of
collaboration amongst staff at member institutions, and to
provide venues for their fruitful exchange of ideas and
experiences. Forums (the DLF has hosted 4 since Fall 1999), the
physical meetings and off-line communication of DLF-supported
working groups (the DLF has supported over 30 such groups), and
DLF publications and progress reports (the DLF has produced 8
printed publications since inception, 4 online Newsletters since
July 2000, and over 40 online reports, again since inception) are
crucial components of this effort.
DLF members participate extensively and inclusively in DLF
forums and working groups. In the meantime, the DLF publishes
prolifically. Details about the level and extent of participation
in DLF activities and about the DLF's publication profile are
available in Appendix 1, sections A1.7. and
Evidence of impacts may become available through the survey
component of the DLF review (Appendix
3). Some evidence is available in evaluations that the
Director's Office has conducted for the DLF forum, in particular.
Those evaluations demonstrate that the forums offer an
unparalleled opportunity for digital library professionals to
learn and exchange information about cutting edge and highly
practical research, development, and operational activities.
Additional evidence is available in the extent to which new DLF
initiatives are beginning to emerge from the casual discussions
that take place at DLF forums and during DLF working group
meetings (DLF initiatives arising from forum discussion include
those on Open Source Software in libraries, on tools for
developing Encoded Archival Descriptions, on technical metadata,
on methods for assessing use of online collections and services).
It is particularly encouraging to see some new ventures involving
partners that had not hitherto worked together [text
The DLF achieve its aims by working in areas of particular
concern to its research library members. From March 2000, the DLF
began working in six such areas, although it is expected that
will change with time.
The goals the DLF set for its work in these areas is set out
in language drawn from the strategic plan, alongside an
indication of the progress the DLF has made achieving these
A1.3.1. Digital library architectures, technologies, systems
Since March 2000 we have not been as active in this area as
one might have expected. This is not because there is no
technical or architectural substance to our recent initiatives;
rather because the technical components of DLF initiatives tend
to be tied up in defining requirements for services that digital
libraries require but cannot develop independently.
The following table supplies our stated goals, and the
progress we have made in meeting them.
|Define, clarify, and develop prototypes for digital
library systems and system components.
||Evidence of progress in well recognized and increasingly
accepted work on key architectural components of
certificate-based authentication, localized reference linking;
architecture of digital repositories; on the OAI Metadata
Harvesting protocol; and on the functional requirements of
infrastructural digital library services such as a registry for
digital reproductions of books and serials, an image distribution
|Scan the larger technical environment for potentially
important trends and practices.
||Not formally active, though most DLF initiatives respond to
|Encourage technology transfer and information sharing
between and among DLF members and between the DLF and appropriate
commercial and industrial sectors.
||Information exchange amongst members happens best at the
Forum, in DLF initiatives and working groups, and through
Newsletter and the web-based reports on DLF initiatives. Work on
localized reference linking and on OAI may potentially transfer
between libraries and industry.
|Communicate technical directions and accomplishments of
the DLF to a wider audience.
||Not a central concern though the DLF website and DLF
publications are publicly accessible and DLF staff and fellows
regularly speak to audiences of non-DLF members and publish in
relevant journals etc.
A1.3.2. Digital collections
|Identify, evaluate and, develop collection strategies and
practices that are appropriate for the digital library and assess
the legal, organizational, and business implications of these
||The DLF has commissioned research and published extensively
in this area, notably in its series "Strategies for building
sustainable digital collections", but also through work on AIC
|Prepare guidelines and other informational materials that
help inform local collection development decision.
||The above publications include such guidelines. In addition,
DLF has developed model licenses, and benchmarks for the
production and documentation of certain kinds of digital
|Encourage the development of new kinds of scholarly
collections that take full advantage of computer and network
||Work on AIC/ArtSTOR focused on collections of digital images.
Work with OAI encourages development of Internet gateways. There
is a much more to be done encouraging digital libraries to gain
maximum advantage from other kinds of digital objects such as
GIS, social science data, audio, film, etc.
A1.3.3. Use, users, user support, and user services
|Engage more effectively with user communities as a means
of building better, more useful, and more useable collections and
||Work on collection strategies (Appendix 1, Section A1.3.2.)
and on methods for assessing use of online collections and
services (Appendix 1, Section A1.3.3.) emphasize the importance
of library engagement with user communities. DLF initiatives do
not typically involve end users except insofar as professional
library staff use the ideas, tools, services, etc. that DLF
initiatives focus on.
|Provide data about users and their behavior in online
environments as a means of informing the development of digital
library collections and services.
||The DLF is helping to extend the amount and improve the
quality of such data focusing in particular on data that are
comparable across institutions. Notable contributions have come
through our study of the information-seeking behavior of
humanities scholars (Palmer and Brockman); our investigation
(with Outsell Inc.) into the nature and use of the scholarly
information landscape at liberal arts colleges and research
universities (forthcoming). We are also working on assessment
methods, notably by documenting effective practices currently in
use through our survey of effective methods for assessing use of
online collections and services. An instructional workshop based
on the survey was held in June 2001 and attended by nearly 45
|Build sustainable user support services that encourage use
and are appropriate to our evolving digital library service
||There is more to be done here. At present our work in this
area extends to support for an investigation by Charles McClure
and David Lankes into benchmarking online reference
A1.3.4. Digital preservation
|Encourage initiatives that result in some practical
experience of digital preservation.
||Work defining and building cross-community consensus around
minimum requirements of e-journal archives contributed to The
Andrew W. Mellon Foundation' e-journal archiving program. The DLF
continues to support the program.
The DLF is also supporting CLIR in its work with the Library
of Congress in framing the national digital preservation program
for which the Library has received up to $100,000,000 from the US
Congress. The DLF's role in this contract will be to help build a
conceptual framework within which a national digital preservation
strategy may be developed.
|Develop and work within a framework that allows such
initiatives to document, compare, and evaluate their
||The DLF maintains the website for the Mellon's e-journal
archiving program and supports its participants in information
sharing and reporting to the community. It will also play a role
in helping to ensure that the learning involved in work with the
Library of Congress is communicated to the broader
|Continuing research and development in the most poorly
understood areas associated with digital preservation.
||Work under the contract with the Library of Congress and the
Mellon e-journal archiving program promises to move us into new
areas including the preservation of electronic books, the Web,
digital audio, etc.
A1.3.5. Standards and best practices
|Encourage greater interoperability and exchange of digital
||The DLF has helped make interoperability and information
exchange more feasible than it has been at any time in the past.
Here one thinks about the DLF's work developing and encouraging
widespread adoption or use of tools and services that
fundamentally lower barriers to interoperability. The tools
include minimum-level agreements (benchmarks) pertaining to the
use of various data and metadata standards; network protocols
(e.g. OAI, X.509). Services include an image distribution service
(ArtSTOR), a registry for digitally reformatted books and
serials, Internet gateways built with the OAI harvesting
There is a great deal more to be done with both services and
tools. But perhaps the greatest effort remains in non-technical
- We lack a common understanding of what "interoperability"
actually entails, or even what it means
- Accordingly, we lack the means of "benchmarking"
interoperable collections and aggregating services
- Institutions are not yet fully comfortable contributing
information content to aggregated collections with which they may
not be obviously associated
- Beyond the large bibliographic databases, we lack services -
even prototypes - that clearly demonstrate the advantages that
cross-collection searching provides to end-users and to
information providers such as libraries, or that can provide
information on how users want to engage with such services.
|Facilitating long-term access to such
|Exercising some influence over the behavior of third-party
||We have had some influence here by organizationally endorsing
particular practices. We have been particularly successful with
localized reference linking solutions, with recommended uses of
TEI in libraries, and with the Liblicense model licensing
agreement. We have high hopes for OAI, benchmarks for digitally
reformatted books and serial publications, and our technical
metadata standards (METS).
A1.3.6. Institutional roles and responsibilities of the
21st-century digital library
|Undertake a survey that provides baseline data about the
organizational, financial, and legal contexts in which digital
libraries are beginning to emerge.
||The survey has been completed and an initial report is
available. Case studies are being developed to enrich that report
and will be included in the final publication (January
|Develop papers targeting university presidents and
provosts of universities and liberal arts colleges, documenting
key digital library challenges and demonstrating their relevance
to university mission.
||With CLIR we have created CLIRinghouse - a periodic
newsletter that has this aim precisely.
|Identify other communities that shape the landscapes in
which digital libraries develop, assess their interests and
motivations, and prepare a targeted literature about the digital
library's significance and its issues.
||CLIRinghouse has this growth potential.
The strategic plan indicates that the DLF uses the following
means to meet its goals and realize its aims
- A central office including a director and a small support
- Distinguished Fellows - senior information professionals who
are supported by the DLF to pursue research or lead catalytic
initiatives in areas of mutual interest.
- An Executive committee that advises the Director on policy
and other strategic matters
- A Steering Committee that acts as the governing body of the
- Informal advisory groups that may be convened by the Director
to advise on initiatives or progress within program areas
- DLF initiatives where shared investigative,
information-sharing, and catalytic activities take place.
- A communications arm including:
- a network of electronic mailing lists;
- a quarterly newsletter which reports the progress of DLF
initiatives to the members and through which, members report to
one another on their own digital library developments;
- online registries comprising information about (a) members'
web-accessible public domain, digitized collections and (b)
policies, strategies, working papers, standards and other
application guidelines, and technical documentation developed by
DLF members to inform or reflect upon their digital library
- bi-annual DLF Forums; and
- publications that report on DLF initiatives and on the
proceedings of the DLF Forum.
With regard to the organizational means at the DLF's disposal,
all work well and by and large as indicated in the strategic
plan. Only the informal advisory groups have never really
materialized, nor is a need for them felt at present.
Where the DLF's communications are is concerned, this is an
essential means for building the community of professionals and
inter-working institutions that is so essential to the success of
individual DLF initiatives and to the organization over all.
In Spring 2001, the DLF Directors Office undertook a survey of
how its various communications efforts were received by DLF
members. The survey results are supplied in some greater detail
in a companion document that lists and assesses individual DLF
initiatives. In general, the survey found that the DLF's
newsletter and its extensive and growing website are well
received, and the DLF forum has evolved as both a major success
and a leading digital library conference.
Details about DLF website use and publication profile are
available in Appendix 1, Section A1.6.
More subjective evidence of the DLF's community-building
progress is evident in the fact that the majority of new
initiatives now originate outside the Director's Office.
A1.5. Summary of activity report of the DLF director for the
period April 1 2000 - July 31 2001
Text omitted from public draft.
A1.6. Summary of DLF website use, January 2000 - July
Text omitted from public draft.
A1.7. Level and extent of
participation in DLF Forums and working groups
The following data are drawn from a database providing contact
details about participants in DLF working groups, forums, and
other events. The database includes basic information about
participants in all DLF forums and in 19 of its working groups. A
list of the working groups is supplied below.
Composition of the 19 DLF working groups for which
participant information is available
|Number of participants in working groups
|Number of distinct individuals
|Number of participants from DLF member institutions
|Number of distinct individuals from DLF institutions
Composition of DLF Forums
|Number of participants in 4 DLF forums
|Number of individuals who have attended 4 DLF Forums
19 DLF Working Groups for which participation information is
- Academic Image Cooperative Advisory Board
- Academic Image Cooperative Working Groups
- The Nature and Use of the Scholarly Information
- Assessing Image Quality
- Developing Sustainable Digital Collections (advisory group)
- Tools for Academic Electronic Publishing
- Registry Services
- Digital Library Programs' Organizational, Funding, and Policy
- Open Archives Initiative (OA) Open Day (US based)
- OAI Planning Meeting
- OAI Steering Committee
- OAI Technical Committee
- Shared Cataloguing Tool for Visual Resources
- TEI and XML in Digital Libraries
- Reference Linking
- Workshop on Social Science Data
- Methods for Assessing Use of Online Collections and
A1.8. DLF publications and
reports since inception
A1.8.1. Printed and online publications
The DLF has produced 8 printed publications since
Strategies for Building Digitized Collections
(September 2001) by Abby Smith
Selection and Presentation of Commercially Available
Electronic Resources: Issues and Practices (August 2001) by
Building Sustainable Collections of Free Third-Party Web
Resources (June 2001) by Louis Pitschmann
Guides to Quality in Visual Resource Imaging (July
Systems of Knowledge Organization for Digital Libraries:
Beyond Traditional Authority Files (April 2000) by Gail
The Making of America II Testbed Project: A Digital Library
Service Model (December 1999) by Bernard J. Hurley, John
Price-Wilkin, Merrilee Proffitt and Howard Besser
Preserving the Whole: A Two-Track Approach to Rescuing
Social Science Data and Metadata (June 1999) by Ann Green
with JoAnn Dionne and Martin Dennis
Enabling Access: A Report on a Workshop on Access
Management (February 1999) by Caroline Arms with Judith
Klavans and Don Waters
The DLF has also produced 4 issues of the DLF
Newsletter (online only), initiated in June 2000
A1.8.2. Progress reports on DLF initiatives (online
The DLF has produced over 40 online reports on its
various initiatives. Reports are grouped below under the program
area in which they appear.
- Open Archives Initiative
- A New Approach to Finding Research Materials on the
Web (July 2000) by Priscilla Caplan - describes the potential
benefits to libraries and their users of the Open Archives
Initiative and the Internet gateways that may be constructed with
- The Open Archives Initiative and Digital Libraries
(January 2001) by Daniel Greenstein - introduces the OAI and
supplies a brief history of the DLF's involvement with it as an
- DLF Evaluation of the Open Arcihves Initiative
(January 2001) by Daniel Greenstein - describes the work the DLF
is undertaking in support of the development of a small number of
Internet gateways through which users will access distributed
digital library holdings as if they were part of a single uniform
- Tools for electronic publishing
- Tools for Electronic Publishing (March 2001) by Maria
Bonn - a discussion paper presented to a meeting of librarians
and scholarly publishers to explore possible shared technical
- Reference linking
- First Workshop on Linkage from Citations to Electronic
Journal Literature. Report of a one day invitational workshop
held on February 11, 1999 (March 1999) by staff at NISO
- Report of the Second Workshop on Linkage from Citations to
Journal Literature. Report of a one-day invitational workshop
held on June 9, 1999 (July 1999) by staff at NISO
- NISO/DLF/CrossRef Workshop on Localization in Reference
Linking. Report of a one-day invitational workshop held on
July 24, 2000 (August 2000) by staff at NISO
- Digital certificates
- A Digital Library Authentication and Authorization
Architecture (March 2000). Describing an architecture,
protocol and operational model for using X.509 digital
certificates for authentication and a directory service to serve
user attributes to determine the level of authorized access to
licensed online materials. The model was developed by the
participants at a DLF-sponsored meeting January 19-20, 1999
- Digital Library Authentication and Authorization
(March 2000?). Describes the DLF initiative to develop an
architecture, protocol and operational model for using X.509
digital certificates for authentication, etc.
- Digital Certificate Infrastructure. Frequently Asked
Questions (1999?). FAQ published jointly with the Corporation
for Research and Educational Networking about the use of digital
certificates, and targeted generally to senior campus
- Distributed finding aids
- Supporting Access to Diverse and Distributed Finding Aids.
A Final Reoprt to the Digital Library Federation on the
Distributed Finding Aid Server Project (July 1999) by John
Price Wilkin. Report on a project that explored the means and
costs of searching encoded finding aids that are distributed at
Preservation of electronic
- Minimum criteria for an archival repository of digital
scholarly journals, version 1.2 (May 2000) by Daniel
Greenstein and Deanna Marcum. Outlines minimum criteria agreed at
invitational workshops of librarians and publishers respectively.
The criteria serve as a touchstone for the repositories being
planned by projects involved in The Andrew W. Mellon Foundation's
e-journal archiving program
- Website of The Andrew W. Mellon Foundation's e-journal
archiving program (from February 2001). Numerous
web-documents introducing the program and describing its aims and
giving details about the individual projects including their
successful grant applications and periodically updated progress
- Registry of digital reproductions of paper-based books and
- Registry of Digital Reproductions of Paper-based Books and
Serials (July 2001) by Dale Flecker and Daniel Greenstein.
Introduces the DLF's work developing a functional specification
and business case for such a registry and facilitating its
- More Access at Less Cost: The Case for a Digital
Registry (July 2001) by Gerald George. Making a case for and
explaining the benefit to libraries of a registry of digital
reproductions of paper-based books and serials.
- Registry of Digital Reproductions of Paper-based Books and
Serials. Functional requirements (July 2001) by Dale Flecker.
Functional specification for a registry of digital reproductions
of paper-based books and serials.
- Draft report of a meeting held on 11 April 2001 to
consider the potential uses of a service that registers digitized
books and journals and to consider implementation (April
2001) by Daniel Greenstein.
- The Academic Image Cooperative
- The Academic Image Cooperative (2000) by Daniel
Greenstein. Introduces the DLF's work on this prototype service
for aggregating and distributing curriculum-based digital images
for courses in art history and other humanities disciplines.
- Final Report of the AIC as submitted to the Andrew W.
Mellon Foundation in August 2000 (August 2000) by Daniel
Greenstein. Traces the progress of the AIC from its initial
conception through the deveolopment of collection, technical, and
image documentation standards, service specifications, and
- The AIC Draft Collection Strategy and Development
Framework (April 2000) by Daniel Greenstein. Resulting from a
review of the AIC's progress during the period 1/99-2/00, the
document outlines a collection strategy and development framework
that may sustain the initiative in the longer term.
- AIC Brochure (January 2000) by Rebecca Graham. This
brief document outlines the AIC's vision and was developed to be
circulated at conferences and other professional gatherings where
the prototype was demonstrated
- Academic Image Cooperative. Reports on initial meetings
held January and May 1999 (June 1999).
- Shared cataloguing tool for visual resources
- Towards a Shared Cataloguing Tool for VR Collections
(December 2000) by Daniel Greenstein. Discussion paper used to
launch an investigation into the functional requirements of and
business case for a shared catalogue of or visual resource
- Draft report on a meeting to explore possibilities for
developing a shared VR cataloguing service... (February
2001). Report on an initial meeting to consider the development
of such a tool/service.
- Shared Cataloguing Tool for Visual Resources (April
2001) by Max Marmor and Sherman Clarke. Report of a focus group
meeting on the functional requirements developed by the DLF for a
shared cataloguing tool for visual resources convened at ARLIS/NA
Annual Conference, Getty Research Institute on April 1,
- Strategies for developing digital collections
- Strategies for developing sustainable and scaleable
digital library collections (May 2000) by Daniel Greenstein.
Document outlining a DLF initiative that aims to assemble,
review, and document practices adopted by libraries in developing
their digital collections. The initiative resulted in several
printed publications of the DLF.
- Social science data
- Digital Library Federation Workshop on Social Science Data
Archives (Feburary 1999). Reports on a meeting convened by
the DLF on the state of the art of digital libraries in the
social sciences to explore current problems and emerging
solutions in three areas: facilities for users to discover and
retrieve relevant and related data sets; means for users to
interpret and evaluate the comparability of data sets; and tools
for methods of data extraction and analysis.
A1.8.6. Standards and good practices
- Benchmarking digital reproductions of printed books and
- Draft benchmark for digital reproductions of printed books
and serial publications (July 2001) by Daniel Greenstein.
Document recommending a minimum benchmark for digital
reproductions of printed book and serial publications and
outlining the importance, rationale, and implications of such a
- Report of a meeting of the DLF on preservation
reformatting practices (July 2001) by Daniel Greenstein.
Report of a meeting at which current practices were analyzed and
- Assessing image quality
- Report of Imaging Practitioners Meeting on 30 March 2001
to Consider How the Quality of Digital Imaging Systems and
Digital Images may be Fairly Evaluated (May 2001) by Stephen
- Technical, administrative, and structural metadata
- The Making of America, Part 2. Introducing work,
supported by the DLF to organize and develop community practices
for creating and encoding the digitized versions of primary
sources and enabling readers to link seamlessly to these
digitized surrogates directly from the finding aid descriptions
of them. The MOA2 project received substantial implementation and
other support from the National Endowment for the Humanities and
the Andrew W. Mellon Foundation.
- Structural, technical, and administrative metadata
standards. A discussion document (December 2000) by Jerome
McDonough. Document used to initiative discussion that led to the
development of the METS standard.
- Report of the Making of America II DTD DLF Workshop
(March 2000), by Jerome McDonough. Report on a meeting convened
at NYU in February 2001 to review experience with MOA2 DTD and to
recommend changes. The report recommended the development of a
new DTD, METS.
- Metadata Encoding and Transmission Standard (METS).
Official Web Site. Maintained by the Library of Congress, the
website supplies overview, tutorial, schema and documentation for
the use of the METS standard that developed out of a
- Licensing commercial content
- Liblicense. Licensing Digital Information. A resource for
Librarians. Web pages maintained at Yale University and
including the CLIR/DLF Model License.
- TEI in Libraries
- TEI Text Encoding in Libraries. Draft Guidelines for Best
Encoding Practices. Version 1.0 (July 30, 1999) Perry
Willetts. Guidelines growing out of a workshop convened to
explore the use of the Text Encoding Initiative (TEI) and XML in
libraries. They make recommendations pertaining to the
application of the TEI Guidelines and particularly "best
practices" for the encoding of electronic texts developed for
different purposes. The guidelines have been endorsed and are in
use by leading text centers in the US and Europe.
- TEI and XML in Digital Libraries. Report of a Two-Day
Meeting held June 30 - July 1, 1998 at the Library of
Congress (1998) by Lee Ellen Friedland and John Price-Wilkin.
Report on the meeting that framed work on the guidelines
- Archival authority control
- Developing a Standard for Recording Contextual Information
for Archival and Manuscript Materials (December 1998). Report
of a meeting at Yale University, New Haven, Connecticut (December
1998) to discuss the establishment of an international
collaborative project to advance the definition and
implementation of a methodology for recording information about
the contexts in which archival records, personal papers, and
similar materials have been created and used
A1.8.7. Use and users
- Methods for assessing use and usability of online
collections and services
- Usage, Usability, and User support (April 200) by
Daniel Greenstein. Report of a discussion convened at the DLF
Forum on 2 April 2000 to frame a DLF investigation into methods
for assessing the use and usability of online collections and
- Assessing changing patterns of library use
- How and Why Libraries are Changing (January 2001) by
Denise Troll. A discussion paper used to focus a DLF initiative
to investigate the extent and use of the scholarly information
landscape at universities and liberal arts colleges, and the
library's contributions to that landscape.
- Dimension and Use of the Scholarly Information
Environment (July 2001), by Lynn Dagar, Daniel Greenstein,
LeighWatson Healy. Proposal outlining aims, goals, methods for
the research that is indicated above.
A1.8.8. Library roles and responsibilities
- Digital library policies, organizations, and practices
- Draft report of a meeting held on 10 April in Washington
DC to discuss preliminary results of a survey issued by the DLF
to its members (May 2001) by D Greenstein, S Thorin, D
Mckinney. Report on a meeting to discuss results of a survey that
was used to help identify the institutional contexts in which
digital libraries are being developed and to create a profile of
the programs and initiatives at institutions that comprise the
Digital Library Federation (DLF).
- Digital Library Policies, Organizations, and Practices.
DLF Survey (January 2001) by Daniel Greenstein and Suzanne
Thorin. Survey instrument used by the DLF to help identify the
institutional contexts in which digital libraries are being
developed and to create a profile of the programs and initiatives
at institutions that comprise the Digital Library Federation
Appendix 2. DLF
initiatives: description, impacts, and costs
Prepared for the DLF's five-year review by
Table of contents
A2.2. Catalytic efforts
A2.3. Standards and best practices
A2.4. Pooled research
From its inception in 1995, the DLF set itself the task of
"creating the means to bring together digital materials - from
across the nation and beyond - that will be made accessible to
students, scholars, citizens everywhere". As it enters its sixth
year, it is settling on a process for achieving this ambitious
aim. In particular, the DLF is demonstrating its ability in:
- incubating new services and tools that digital libraries
commonly require but cannot independently develop or
- identifying, endorsing, and promoting the standards and good
and best practices that support the digital library's interests
in high-quality, interoperable, and sustainable online
collections and services;
- pooling members' research and development effort for common
- facilitating communication amongst DLF members and ensuring
that knowledge gained by the DLF is transferred into the broader
This document lists initiatives that the DLF has sponsored
since its inception and includes initiatives that are planned but
not yet formally begun. [Cost indications have been omitted from
the public draft].
Initiatives are grouped with reference to the DLF's aims
- Catalytic (work incubating services etc)
- Work on standards and best practice
- Pooled research
DLF funding stimulates the development of so-called
infrastructural or utility tools and services that are commonly
required by digital libraries but beyond their independent means.
Work in this area develops functional specifications and business
cases for the development of these tools or services. Working in
this way, the DLF seeks to reduce the risk to investors who have
the financial capacity to develop such tools and services. In
this respect, DLF investment may be seen as incentive or venture
||Description and impact
|Academic Image Cooperative
January 1999 - August 2000
|The DLF's work prototyping organizational, business, and
technical aspects of an image distribution service contributed
directly to the formation and launch recently by the Mellon
Foundation of ArtSTOR - "an independent not-for-profit
organization that will develop, 'store', and distribute
electronically digital images and related scholarly materials for
the study of art, architecture, and other fields in the
humanities". ArtSTOR marks a major advance in the development and
dissemination of visual image resources that support research and
teaching. [Text deleted]
The DLF continues to work closely with ArtSTOR.
|Shared cataloguing tool for visual resources.
January - April 2001
|ArtStor has also subsumed the DLF's work on a tool to reduce
redundant effort involved in cataloging visual resource.
Beginning in January 2001 the DLF developed an outline functional
specification and market assessment for the cataloguing tool
|Open Archives Initiative (OAI).
March 2000 - October 2002
|Through its two-year investment in the OAI, the DLF has
helped to develop a network protocol upon which the next
generation of scholarly Internet portal services is likely to be
built. With CNI and the NSF, the DLF helps to support:
- the OAI's steering committee (its governing body - the DLF
director is its chair);
- the OAI Executive (located at Cornell University)
- the OAI technical committee
The DLF has also facilitated the development of a number of
Internet gateway services, funding for which is being supplied by
the Andrew W. Mellon Foundation [text deleted].
The OAI is having significant impact in the US and abroad. It
is being evaluated by RLG and OCLC as a means of exposing
bibliographic records to the world wide web (something that will
fundamentally change bibliographic provision and library
service); it is a building block of the NSF's National Science
Digital Library; it is being evaluated in the UK as a means of
integrating access to higher education information services; it
is widely adopted by scholarly preprint repositories in the US
and abroad; it is being considered by some publishers as a means
of revealing. In 2002, the OAI will be developing an exit
strategy and the DLF will not expect to support its activities
after October of that year.
|Minimum requirements of digital archival repositories for
electronic scholarly journals.
January 2000 -
|In early 2000, CLIR, the DLF, and CNI initiated a process
through which they developed consensus amongst publishers and
libraries around minimum requirements of an e-journals digital
archival repository. That consensus helped encourage The Andrew
W. Mellon Foundation to launch its e-journals archiving program
and fund seven institutions [text deleted] to plan the
development of such repositories. In a second phase of the
program to be launched in 2002, it is hoped that some of the
planning projects will be funded to move to an implementation
With this program, libraries and publishers take a significant
and practical step forward in addressing their shared
preservation concerns and developing repositories for e-journals.
The DLF remains closely involved in the Mellon e-journals
archiving program, hosting its web pages and facilitating
discussion and information sharing amongst its participants.
|Localized reference linking.
February 1999 -
|With NISO and other organizations, the DLF has developed a
reference linking architecture that will enable a university
library to resolve reference links to text documents to which
they have access. By endorsing this "localized linking"
architecture, and promoting it to other library associations
(e.g. ARL, ICOLC, etc.), the DLF was also able to encourage its
evaluation (and, we hope, ultimate )adoption by CrossRef, an
organization of some 30+ publishers that is implementing a
reference linking service involving its members' numerous journal
The adoption of this localized linking architecture will
benefit libraries economically by allowing them to benefit from
reference linking systems without constraining their choice of
supplier for citation and full-text content.
|Registry for digitized books and serials
April 2001 -
|An increasing number of libraries and commercial entities are
involved in converting existing paper-based books and serials to
digital form. Unlike the special collections materials that have
been the focus of digital conversion in many libraries, books and
serials are commonly duplicated in many different institutions.
This presents both an opportunity and a threat. The opportunity
is for coordination between institutions, with the efforts of
each contributing to a larger shared but distributed collection.
The threat is that resources will be wasted in the repeated
digitization of the same material.
A key requirement to realizing the opportunity and avoiding
the threat is a mechanism for sharing information in a coherent
fashion between institutions about what has been digitized; that
is, the creation of a Registry of digitized materials.
The DLF has developed a functional specification and business
case for this registry and is currently in discussion with
potential service providers who are positioned to supply it.
The purpose of the Registry is to provide a place for
institutions that have created (or are otherwise responsible for)
digitized versions of traditional printed books and serials to
- what specific items have been (or are about to be)
- where they can be accessed;
- the specifications followed in digitization.
Such a registry promises numerous benefits to libraries,
including the reduction of duplicative effort and in enhanced
access to digitized books and serials.
|Tool for the construction of benchmark quality Encoded
Archival Descriptions (EADs).
Kick-off meeting is scheduled for autumn 2001
As more archival finding aids become available for online
exploration, scholarly and curatorial communities anticipate
being able to search seamlessly and easily across them. Perhaps
the single greatest obstacle to realizing this vision, is the
enormous variation in the way that archival finding aids are
defined and created. This variation persists even as the Encoded
Archival Description (EAD) emerges as the preferred encoding
scheme for archival finding aids. Quite simply, there is
insufficient agreement about how beset to implement the EAD to
create genuinely interoperable online finding.
With this tool, the DLF hopes to encourage adherence to some
minimum-level benchmark requirements and thus to support the
broader aim of developing interoperable finding aids.
The DLF will define the functional requirement of this tool
and facilitate its development by an appropriately positioned
service provider. The work builds directly on the DLF's successes
defining a functional requirement and business case for a visual
resources cataloguing tool, and in finding a service provider to
develop that tool as a production service.
|Shared digitization service
Kick off meeting scheduled for autumn 2001
|The DLF is developing the functional specification and
business case for a service that reliably produces digital
surrogates for items in general and special collections. It will
then facilitate the development of such a service by some
appropriately positioned third party.
Such a "shared digitization service" would allow libraries
large and small to undertake digitization programs without having
to rely so heavily on (a) commercial entities that may not well
understand library needs or (b) in-house production units that
are typically costly, small in scale, and financially
|Repository for heterogeneous digital objects
Kick off meeting scheduled for October 2001
|The DLF is supporting an initiative in which several
institutions will develop prototype digital repositories using a
common repository architecture that has been developed to
coherently and consistently manage large-scale collections of
deeply heterogeneous digital objects (e.g. digital images,
digital texts, digital sound recordings). DLF support for the
initiative runs to an initial meeting of its participants and is
intended in part to assist the initiative in attracting
substantial external funding as required for full implementation.
The meeting is planned for October 5-6, 2001. I is premature
to assess its outputs and impacts.
|Repository and alerting service for open source digital
Kick off meeting scheduled for October 2001
|The DLF is supporting a meeting to investigate what steps it
might take to ensure that its members (and digital libraries more
generally) gain more advantage from increased development in
libraries and library systems suppliers of open source software
The meeting is planned for October 5-6, 2001 so it is
premature to assess its outputs and impacts.
A2.3. Standards and best
Leveraging its members' collective influence the DLF
identifies, endorses, and promotes those standards and good
practices that support the digital library's interests in
high-quality, interoperable, persistent, and sustainable online
collections and services. It is also developing the network of
strategic alliances it will need to ensure that its good and best
practice recommendations are maintained and communicated to the
broadest possible community.
Although there is an opportunistic element in the way that
priorities are assigned to work in this area, effort increasingly
grows out of the best practice requirements that surface in
developing functional and business cases for essential utility
tools and services (see Appendix 2, Section
||Description and impact
|Liblicense model agreement
- June 2001
(The model license was endorsed by the DLF in January
|This model license effectively documents preferred and good
practice and acts as a decision tool that is likely to save
libraries both time and money in negotiation robust contracts
with commercial content providers. Ann Okerson (Yale University)
will maintain the model license so that it evolves to reflect new
circumstances as they arise.
The model license helps to introduce consistency and economy
of effort in library licensing practices and has been taken up
widely by both libraries and publishers in the US and abroad.
|Structural, technical, and administrative metadata for
digital objects (Making of America 2, and METS).
MOA 2 September 1996 - December 1999
METS Feb - Nov 2001
|DLF has supported work on mechanisms for describing
technical, structural, and administrative characteristics of
digital objects. Initial recommendations about technical,
structural, and administrative metadata emerged from the Making
of America II initiative (see
In recent months, these recommendations have been refined and
extended by a DLF working group into a more inclusive "Metadata
Encoding and Transmission Scheme" (METS). Work of the initiative
is fully documented on a website that is being maintained by the
Library of Congress (http://www.loc.gov/mets/. The site includes
an overview and tutorial, a beta version of the METS schema (in
XML), an example METS XML document, and METS documentation.
A common means of recording attributes of digital objects is a
fundamental building block of any distributed digital library
service. It is as essential to institutions that wish to
integrate access to or management of their own distributed
digital collections. Widespread adoption of the METS standard as
it evolves will therefore have a significant impact.
|Benchmarks for digitally reformatting printed books and
June 2001 -
|Libraries and others are digitizing increasing quantities of
printed material for online access without agreement on any
desirable level of imaging quality. The DLF is working to
identify, and build support for, specifications acceptable as the
minimum necessary for digitally reproducing printed books and
serial publications with fidelity.
Adoption of such benchmarks would help users and libraries
both. Users could have more confidence in the fidelity of digital
reproductions available to them. And libraries could produce and
maintain reproductions with confidence that expensive
re-digitization would not become necessary. Digital reproductions
meeting at least the benchmarks' minimum specifications would
remain viable even as reproduction techniques improved. Also,
because such texts would have well-known, consistent properties,
they could support a wide variety of uses (including uses not
possible with printed texts). Additionally, agreement on minimum
benchmarks for digital reproductions of printed publications is
an essential first step for libraries that wish to investigate
whether they could manage and preserve print materials more
effectively if they relied more heavily on digital reproductions
|Methods for evaluating digital image quality
March 2001 -
|Significant investments are being made by cultural
organizations in the digitization of the pictorial collections
that exist in libraries, archives and museums. The activity is
highly decentralized. In the absence of guidelines or best
practices for image digitization, one concern is whether the
resulting digital collections will be interoperable across
curatorial domains: are images that are made in libraries,
museums, and archives, being made in formats that are easily
distributed; to a baseline level of quality that meet users'
In March 2001, the DLF sponsored a meeting in Boston to
establish a forum for expert practitioners to exchange ideas
about what is "good" and if possible to prioritize where tools,
applications, and training will be of greatest benefit to
institutions interested in making digital reproductions of
consistent quality and persistent utility.
The forum continues and is pursuing a number of practical
investigations of its own design (e.g. into the impacts of
digital libraries of JPEG 2000). It has also been invaluable to
the DLF as a source of expertise on other benchmarking issues
that arise in consideration of digital image content.
|Implementation guidelines for the Visual Resources
Association Core Categories version 3.0
September 2001 - September 2003
|The Visual Resources Association has developed an outline
scheme for describing works of art. What the scheme lacks is
implementation guidelines that will encourage visual resource
cataloguers to develop visual resource descriptions that conform
at least along minimal lines.
The absence of such guidelines emerged as a critical obstacle
to the development of a shared cataloguing tool for visual
resources and even in the development and deployment of an image
distribution service such as ArtSTOR.
DLF support for the development of these guidelines has helped
to attract matching support from the Getty Research
|Benchmarks for digital archival repositories for electronic
January - September 2000
|In early 2000, CLIR, CNI, and the DLF initiated a process
through which they developed consensus amongst publishers and
libraries around minimum requirements of an e-journals digital
archival repository. That consensus helped encourage The Andrew
W. Mellon Foundation to launch its e-journals archiving program
in which seven institutions [text deleted] are planning the
development of such repositories.
|Benchmarking digital reference services
August 2001 - August 2002
|As the number of digital (online) reference services grow in
library and commercial communities, so does the need for some
means of assessing their value, quality, and performance. As
library patrons demand more services online, and as reference
librarians seek to better meet patrons' information needs through
the Internet, it has become essential to determine common
definitions of success and quality. Library administrators need
strong, grounded metrics and commonly understood data to support
digital reference services, assess the success of these services,
determine resource allocation to services, and determine a means
for constant improvement of digital reference within their
For these reasons, the DLF is working together with OCLC, the
Syracuse University, Florida State University, the Library of
Congress and others to develop methods to assess the quality of
digital reference services, test and refine measures and quality
standards to describe digital reference services, and to produce
a guidebook that describes how to collect and report data for
these measures and standards.
|TEI text encoding in libraries.
June 1998 - July 1999 (guidelines endorsed by the DLF in
|In the early 1990s, the Text Encoding Initiative prepared
guidelines for the use of SGML for encoding texts. They did not,
however, propose implementation guidelines. Accordingly, as the
number of TEI-encoded texts grew, so did the variations in the
application of the TEI encoding scheme. Ironically, the absence
of implementation guidelines or minimum requirements impeded the
TEI from achieving its key aim: encouraging the interoperability
and exchange of encoded texts.
To address this situation, the DLF supported the development
of guidelines pertaining to the application of the TEI and
particularly "best practices" for the encoding of electronic
texts developed for different purposes.
The guidelines have been endorsed by the DLF and are in use by
leading text centers in the US and Europe.
A2.4. Pooled research
The DLF pools research and development expertise at member and
other institutions to survey and evaluate existing practice and
to answer pressing strategic, organizational, and technical
questions. Work in this area is driven by members' perceived
needs, by opportunities that are available to conduct meaningful
research, and, increasingly, by the questions arising from DLF
investigations into infrastructural tools and services (see
Appendix 2, Section A2.2)
||Description and impact
|Strategies for developing sustainable digital collections
April 2000 - July 2001
|The development of useful and sustainable digital collections
is one of the greatest single challenge confronting digital
libraries. Whether the digital information comes from a
commercial publisher or from a digitization unit within the
library, it seems to exist under a cloud of profound and
unsettling uncertainty. Will it be useful and useable in its
present or intended form, or require additional work on the part
of catalogers, systems staff, or subject bibliographers? What new
demands will its availability make on library reference staff?
What level of continued investment will be necessary to ensure
its accessibility on current hardware and software?
In April 2000, the DLF commissioned three reports to address
these questions basing their analysis on the substantial
experience that leading research libraries are amassing with
their digital collections. The reports deal respectively with
commercial electronic content, digital materials created from
library holdings, and Web-based "gateways" that link to selected
Internet resources in the public domain.
Adhering to a common outline, the reports demonstrate how
decisions taken by a library when acquiring (or creating)
electronic information influence how, at what cost, and by whom
the information will be used, maintained, and supported. By
assembling and reviewing current practice, the reports aim where
possible to document effective practices. In most cases, they are
able at least to articulate the strategic questions that
libraries will want to address when planning their digital
The reports are only now being published so it is premature to
judge their impact. Certainly, response to public presentations
based on earlier drafts has been very promising.
|Digital library policies, organizations, and practices.
January 2001 - January 2001
|The DLF has completed a member survey designed to identify
the institutional contexts in which digital libraries are being
developed. By documenting the very different paths along which
digital libraries evolve, and the different ways they organize
and fund themselves, the survey will: inform strategic planning
and decision making within digital libraries; provide benchmarks
for assessing digital library development; and identify emerging
A draft report based on the survey data is available from the
DLF website. A formal report that will contain case studies that
document the key digital library development paths that are
apparent in the data will be available at the end of the calendar
year. Accordingly, it is premature to assess the initiative's
|The library's role in the broader scholarly information
||In March 2001 the DLF convened directors from eight academic
libraries to consider a white paper on how and why library use is
changing, and to discuss how DLF could enhance and extend efforts
to assess and then understand the likely impacts of those
changes. Given the emphasis by ARL and others on developing
methods for measuring the provision and impacts of library
collections and services, it was agreed that the DLF might focus
on broader scholarly information landscape to which those
collections and services contributed one part.
In response, the DLF is working with Outsell Inc. on a study
that will reveal the extent, nature, and use of that scholarly
information landscape at leading liberal arts colleges and
research universities. This knowledge will be invaluable for
libraries and universities in planning information services to
focus explicitly on the current and emerging needs of their
faculty and students, and to avoid focusing on what is not, or
may no longer be, important. The academic community will also
benefit as publishers and content providers that serve the
education market create better information products based on an
increased knowledge of users' needs.
When the study is completed, the DLF will follow up with some
of the libraries whose directors were initially convened to shape
the study. Working closely with those libraries, it will examine
how the study's results help to interpret local library trends
(for which data exist, for example, through ARL and other bodies)
and how survey results might or should impact upon library
DLF has commissioned the study design from Outsell Inc. and is
seeking external funding to support implementation. Both the
study design and a funding decision are expected on 9/15/01.
|Methods for assessing use of online collections and services
January - October 2001
|How to assess the use and usability of online library
collections and services is a significant challenge that
confronts DLF members and other leading digital libraries. Such
assessment is critical for evaluating the impact of online
activities. It is as crucial for library planning.
Though nowhere formally documented, leading research libraries
have a wealth of relevant assessment experience. Surfacing that
experience is the aim of a survey that is being conducted by DLF
Distinguished Fellow Denise Troll (Carnegie Mellon University).
Ms Troll's survey looks at assessment practices at DLF member
institutions and is based upon 65 interviews. It summarizes,
documents, and assesses current practices, highlights lessons
learns and focuses on practices that have proven to be
particularly effective. The survey report will include
illustrative case studies. Although it is not intended as an
tutorial in assessment methods, it will supply methodological
overviews and provide extensive pointers to those who need to
A draft of the study has been completed. Final publication is
expected in Fall 2001.
|Issues in electronic scholarly publishing
|In May 2001, the DLF convened a small group of
representatives from university libraries and university presses
in order to:
- discuss their different approaches to electronic scholarly
- establish a venue for information sharing about appropriate
tools and technologies, and
- encourage collaborative developments.
The meeting was intended as a first step in an exploration of
production and distribution needs that may be common to libraries
and university presses and as such met through some common
To date, no concrete results have materialized from the
|How humanities scholars use and value library information
||A study commissioned by the DLF into how humanities scholars
locate, explore, and use research materials in print and digital
The report (to be published in Fall 2001) is likely to have
profound implications for how libraries prioritize investment in
online collections and services.
Although it is too soon to assess the study's impact,
preliminary findings were presented to the DLF Fall Forum 2000 to
|Outsell research and information service .
||The DLF subscribes to this information service.
Weekly reports are circulated to DLF members. Value of these
reports has yet to be assessed.
||Under DLF auspices, the California Digital Library, Columbia
University, JSTOR, and OCLC developed a protocol that enables an
information resource provider to verify that a user bearing a
digital certificate has authority from a home institution to use
a requested resource. The prototype system combines the use of
X.509 digital certificates for authentication with a directory
service providing authorization to licensed resources based on
The work produced an architecture statement and
contextualizing material and has been very influential in shaping
direction of research and practical experimentation with
certificate based authentication systems including work
undertaken by Intenet2.
|Working with distributed finding aids.
||Report commissioned by the DLF on the means and costs of
searching encoded finding aids that are distributed at different
|Workshop on social science and government data
||In January 1999 the DLF supported a workshop hosted at
Princeton University to explore common needs as felt by social
science data managers and other experts into: the discovery and
retrieval of databases, the evaluation and interpretation of
alternative data sources, and data extraction for analysis and
Workshop participants identified a number of concrete next
steps in a report that provides a useful survey of challenges and
opportunities in the social science data community.
No follow up work has been conducted.
To build a community of professionals, the DLF facilitates its
members' communication, collaboration, and cross-fertilization.
Early indications from an evaluation instrument recently
circulated to staff at member institutions suggest the DLF's
newsletter (now published twice rather than four times a year)
and its extensive and growing website are well received.
||Description and summary of survey response
|DLF mailing lists.
||The DLF uses an "announce list" to communicate with staff at
DLF member institutions about its initiatives, publications, etc.
The list, with 96 members (126 when DLF directors are
included) is well regarded by members who have asked us to
advertise its existence (and methods for joining) more widely. We
are also encouraged not to use the list to circulate information
and announcements originating outside the DLF.
|The DLF Web site.
||The Web site was comprehensively redesigned and reorganized
in Spring 2000 to anticipate growth in the number of DLF
initiatives and accommodate new reporting mechanisms (e.g. the
Newsletter, the registries, the regular reports from the
The Web site is well consulted and generally found useful and
navigable. Nonetheless, respondents emphasized the need for the
DLF to alert people to new additions to the site as they are
posted. And respondents made several specific suggestions for
improvement of the site that will be taken up over the next
|The DLF Newsletter.
||The DLF Newsletter includes regular reports from member
institutions on their digital library developments.
It is well received and will continue to be published twice a
year (requiring annual reports from DLF member institutions) and
members will be notified of its "publication".
|DLF printed publications.
||These are the edited and highly polished reports that result
from DLF-sponsored initiatives. The reports have been integrated
into CLIR's "burgundy series" lending them credibility and, we
hope, strengthening the series as a whole.
A "communications survey" commissioned by CLIR found that the
publications are very well received and regarded.
|DLF Directors' reports.
||These regular progress reports appear in the Newsletter and
summarize progress of DLF initiatives.
The reports are generally well received. In future, attention
will be paid to developing bulleted summaries that emphasize the
significance of the DLF's various projects, and notifying members
of new reports as they are posted.
|DLF project reports.
||These more detailed reports chart work in progress and
summarize initiatives with results that do not lend themselves to
publication in the "burgundy series."
Project reports are particularly well appreciated by project
participants (they help to track and focus effort). They are also
well received by other specialists who find their way to the
reports out of interest or need. They are not often read by
people who come to the DLF website with a more general set of
interests. The generalists who do read these reports find them to
be too detailed or too technical. The DLF will continue to
publish these reports on its Web site and will take steps to
ensure that they attract and are readily accessible to the
appropriate, specialist audience for which they are intended.
||The DLF has three registries: a calendar of digital library
events sponsored by the DLF or member institutions; a
web-searchable database of policies, strategies, working papers,
standards and other application guidelines, and technical
documentation developed by DLF members to inform or reflect upon
their digital library development; and a web-searchable database
of members' public-domain, online digital collections. A majority
of people who responded to the communications survey do not
consult the registries, and of those who do, few find them
particularly useful. In light of these comments it is proposed to
discontinue the events calendar and seek means of promoting and
encouraging use of the digital collections registry and the
|The DLF Forum.
||The Forum is hosted periodically to enrich, stimulate, and
inform the work of staff at DLF member institutions and to
showcase and gain input into current DLF initiatives.
A survey conducted after the Fall 2000 Forum fall suggests
that the forum is a great success and fills an important void for
information professionals actively engaged in the development of
digital libraries. The survey also suggests that the forums
should: continue for the present on a twice-yearly basis; take
place over two full days; provide ample opportunity to discuss
and develop current DLF initiatives; provide opportunities to
hear from representatives of funding agencies and other bodies
that exercise some strategic influence over or leadership in
digital library development.
September 3, 2001
Conducted by: The Communications Office, Inc.
108 East Del Ray Avenue
Alexandria, Virginia 22301
Revised by the DLF Review Panel to Protect Confidentiality
Table of Contents
A3.1. Survey of DLF Steering Committee:
Findings and Highlights
A3.2. Survey of Key Informants: Findings and
A3.1. Survey of DLF
Steering Committee: Findings and Highlights
- Why are you a member of the DLF?
The theme for many was
"collaboration." There appears to be an overarching philosophy
that the sum of the work of the DLF is greater than its parts.
Members are eager, especially for their staff members, to share
information and problem solving. With the tremendous volume of
work to be done, there needs to be economy of effort among
members. The majority of respondents see their institutions as
contributing strong currency to the collaborative process that
will benefit the "whole." A few talk about what DLF can provide
to their individual institutions. It is important to members that
the DLF's work fill a unique niche and not be redundant in areas
addressed by other organizations. Many Steering Committee members
talked about the DLF's value to their staffs' development.
- Has the DLF enabled its members to accomplish what could
not have been done individually or informally?
The unanimous response to this
question is "yes."
- What has the DLF enabled its members to accomplish?
Once again, the opportunity to
collaborate was a central theme. More specifically, the Open
Archives Initiative metadata harvesting study and facilitated
access to grant money were often mentioned. Among the other
responses were cross reference mobilization, survey research,
identification of best practices and models, metadata standards,
preservation of electronic journals, registry of digitized books
and serials, the academic image cooperative (ArtSTOR), and
standards work such as METS.
- Has your institution profited from its investment of time,
money, and attention in the DLF? What value has it received?
Every institutional member said
yes, they have profited from their investment in the DLF. Again,
they noted the value of collaboration and facilitated access to
grant money. There is no doubt that the DLF forums are highly
valued. At this point in the survey, a few comments emerged
indicating that some Steering Committee members believe the
collaborative value of DLF rests more with university staff
members who are directly involved in digital programs, not with
the Steering Committee members. This theme recurs more strongly
further into the survey.
- How can the DLF better assist you in leveraging your
The spectrum of responses was
broad, as indicated in the following sample responses. Once
again, there were comments urging more inclusion, in terms of
library staff from DLF institutions. Members want DLF to take a
stronger, more visible leadership role on digital issues. The
belief is that with strength will come more funding opportunities
for the organization and its members. They would like to see
their investment in the DLF reap more grant dollars.
- Has your rationale for belonging to the DLF changed over
the years? If so, how?
About half of the respondents said
their rationale for belonging to the DLF has changed. Many said
their expectations for DLF have changed positively as the
organization has matured. Once again, the subject of giving
library staff a more active role in the DLF was discussed by a
few. There appears to have been a shift in mindset as members
initially set out to create one digital library; whereas today,
they are working on a collaborative model built on multiple
institutional contributions. There is concern expressed by a few
that, as DLF has grown, its power and energy have been diluted by
members who do not have the passion and expertise upon which the
organization thrived initially.
- What has the DLF accomplished as an organization?
Many people listed specific
accomplishments, such as a registry for digital objects,
reference linking, standards for digitization, archiving, best
practices, and work user studies, among others. Providing
opportunities for collaboration, bringing visibility to digital
libraries and their issues, and attracting funders were more
broadly defined accomplishments. Yet, others felt accomplishments
were still pending and that more emphasis should be placed on
- In what areas, if any, has the DLF developed a leadership
Members view leadership from many
perspectives. Some were quick to point out specific initiatives--
digitization, open archives initiative, metadata standards and
metadata harvesting, and most notably, best practices. Others
judged DLF's leadership status by prominence in government and
funding agency meetings. Still others believe its leadership is
best characterized by its catalytic ability to inspire
collaboration and novel thinking. One member took exception to
the term "leadership," preferring to cite "success" in providing
positive outcomes to benefit scholars.
- Should the DLF continue?
The answer was a resounding "yes."
One member said "maybe," adding "I think it needs regular
reviews. If it is not accomplishing stuff, it should go out of
- Should the DLF operate at its current level of
There was no consensus. Some
members expressed a desire to better understand DLF's future
direction before they could provide a definitive answer. Others
approached the question from their institution's ability to pay
higher dues, as opposed to assessing the best interest of the
DLF. Of that group, some said they would not be willing to pay
more; a few said they would be happy to pay more if the increase
reaps greater benefits; still others thought an increase might be
warranted, but it should be absorbed through additional members.
Also, it is interesting to note that some equated an increase in
funding with a larger organization, as opposed to expanded
- What funding sources should support the DLF?
Members-to varying degrees-believe
the current membership dues structure, combined with foundation
and government grants, is appropriate. Other suggestions included
commercial sponsorships and creating an entrepreneurial
consulting venture for DLF.
- Should the DLF remain a limited-term organization, or
should it establish itself as a permanent organization?
Most of the respondents believe DLF
should remain a limited-term organization.
- What is the optimal size for the DLF?
There was no consensus on optimal
size. In distilling the content of the various responses, it
would be accurate to say most members believe the current size is
manageable (although a few feel that 26 is too large). However,
there are those who feel the DLF should be more inclusive and
open membership to an unlimited number of institutions that are
able to meet the organization's high standards and dues
requirements. These particular respondents are not necessarily
advocating for a large DLF membership; they believe that the high
entrance standards and financial commitment will effectively
regulate the size of the membership.
Others were more concerned about the quality than the actual
number of members. This topic was introduced earlier in the
This question elicited talk about the potential of changing
organizational structure to accommodate the present membership
and, definitely, a larger one. That topic is addressed in greater
detail further in the survey.
- To whom should the DLF expand its membership?
- What institutions, if any, should DLF expand its
- Is the DLF's current organizational and governing
structure effective in meeting the work and leadership
requirements of the organization?
About half of the respondents
believe DLF's current organizational and governing structure is
effective. However, many of them provided a qualified "yes." The
creation of an Executive Committee to make quick decisions and
keep the organization nimble has been welcomed. There are
concerns about the function of the Steering Committee. Some say
it is ineffective because of its size; others say it is
ineffective because its members are only peripherally involved in
the day-to-day, hands-on operation of digital library programs.
The theme of engaging more project people in the DLF emerges
- What improvements, if any, might you suggest in DLF's
current organizational and governing structure?
The Steering Committee continues to be a focus of concern for
some, in terms of its personality and effectiveness. There is a
feeling that some of the library directors do not have the
expertise or interest to stay actively engaged. This points back
to the contention by some that the Steering Committee members
should address DLF administrative and broad programmatic issues;
those with specific digital library expertise should engage one
another and spearhead collaborative efforts.
- What has been the most successful part of the DLF
As one can see from the following
comments, members see a number of DLF successes. The overwhelming
response is the DLF forums, followed by the Open Archives
Initiative. It is obvious, reviewing these responses, that
members find opportunities to engage their staffs in the work of
DLF to be highly desirous.
- What has been the least successful part of the DLF
Once again, issues of communication
and Steering Committee dynamics were discussed. Other topics were
mentioned-limited publication dissemination, lack of strategic
planning, disconnects between endorsements and implementation,
borne-digital vs. retrospective priorities, funding brokering,
etc. However, relative to the comments provided in the rest of
the survey, this "least successful" question yielded a limited
number of answers.
- What might the future emphasis of the DLF be?
Members have such a diverse "wish
list" for DLF that it is not possible to identify particular
trends. The responses speak for themselves.
- Does the DLF need other collaborators to achieve its
Most respondents see the necessity
for collaborators, but they are quick to point out that the DLF
membership should remain as it is. They do not see collaborators
as future DLF members. A few members are willing to explore
collaborative relationships with commercial vendors, but a
general distrust of that group was expressed. Yet, others express
a desire to at least stay abreast of commercial developments that
could benefit the DLF.
A3.2. Survey of Key
Informants: Findings and Highlights
- Have DLF initiatives advanced the work of the digital
All respondents answered
- How has the DLF advanced the digital library community's
Most respondents answered in the
broad sense of providing a forum for collaboration, setting
standards, conceptualizing and articulating necessary agendas,
promoting staff development, infrastructure development, etc.
- Has any DLF work influenced how you've developed your
digital library program?
This question was not relevant to
all respondents. University librarians cited influence in the
area of standards, infrastructure issues, and shared developments
and approaches. [Text deleted.]
- Has the DLF enabled 'the community' to accomplish what it
could not do individually or informally?
The consensus was "yes," but with
- Should the DLF continue?
Of the eight respondents, only five
gave an unqualified "yes."
Who is The Communications Office?
Digital Information issues are not new to The Communications
Office. It recently conducted a similar survey for the Council on
Library and Information Resources (CLIR). The firm counts among
its clients the Library of Congress, Congressional Research, and
National Archives & Records Administration. It wrote the
Digital Library Federation's organizational brochure ("Chaos and
Revolution") under the direction of Don Waters as well as the
collateral materials for CLIR's "Into The Future" digital memory
crisis campaign in 1998.
return to top >>