CHAPTER 1
INTRODUCTION TO THE STUDY AND FINAL REPORT
At its first meeting in December 1995, the Government Information Locator Service (GILS) Board approved a recommendation by John Carlin, Archivist of the United States, for an evaluation study of GILS. Between September 1996 and March 1997, the investigators conducted extensive data collection and analysis to assess the current status, use, and user satisfaction with the U.S. Federal implementation of GILS. This document is a report of the evaluation study, including findings from the study and recommendations for improving the U.S. Federal GILS initiative.
1.0. THE EVALUATION STUDY
The evaluation of the Federal government's implementation of GILS reported here had as its primary purpose the collection and analysis of information that would lead to an understanding of how:
The Archivist, in his proposal for an evaluation, emphasized the importance of understanding how well "GILS is meeting user information need." He recommended that an evaluation study be conducted that "focuses on who has been using GILS, how well their needs have been served, and what, if any modifications are needed to improve service to the public" (Report of the Initial Meeting of the Government Information Locator Service Board, 12/6/95; See Appendix A–5). The GILS Board established a committee to plan the evaluation.
The General Services Administration contracted with the investigators to conduct the evaluation. Five Federal agencies contributed to the funding of the study: Department of Commerce, Department of Defense (DoD), Environmental Protection Agency (EPA), General Services Administration (GSA), and National Archives and Records Administration (NARA). The Office of Management and Budget provided support for the study through Peter Weiss as the Contracting Officer’s Technical Representative (COTR) Lisa Weber from NARA served as co-COTR. The investigators established a project advisory group, and members of the original evaluation committee served on the advisory group.
The investigators designed and executed a user–based evaluation study that responded to the Archivist’s recommendation. The "users" of a government information locator service, however, are not cut from a single cloth, and the investigators accounted for the perspectives of many "user groups," each with a special interest in the U.S. Federal GILS. A user–based approach sensitizes researchers to multiple stakeholders and users with differing needs and expectations, and the effects of these on assessments of programs and services.
The power of a user–based evaluation is its focus on people—their needs, their expectations, and their assessments. User–based evaluations are aligned with user–based design, where the assumption is not If we build it, they will come nor that assumption’s attendant focus on "systems" and "resources." Instead, user–based design and evaluation focuses on user needs, their behaviors, their requirements, and their assessments of the usability and utility of particular systems and services. While this evaluation study also examined "systems," "resources," and other aspects of the U.S. Federal GILS initiative such as policy and management, users provided a key perspective.
In addition to the extent which GILS implementations by Federal agencies meet the expectations of users, the investigators defined a number of study goals:
The guiding principle for the study was identifying refinements and improvements to the U.S. Federal GILS efforts rather than on measuring strict compliance to policy requirements and technical standards.
The investigators were commissioned to conduct an "Evaluation of the Federal Government’s Implementation of the Government Information Locator Service (GILS)" according to the Statement of Work in the General Services Administration’s Request for Proposal, KECI–96–006 and based upon the Technical Proposal (Moen & McClure, 1996a) submitted in response to the Request for Proposal. The Statement of Work identified specific requirements for the study.
Based on the Statement of Work, the investigators developed the Work Plan (Moen & McClure, 1996b) that detailed study activities and time tables. The project advisory group identified above reviewed the Work Plan, and the COTR accepted the Work Plan as the first deliverable of the study in September 1996. During the study, the advisory group reviewed and provided comments on a progress report (Moen and McClure, 1997), draft preliminary findings and recommendations, and the complete final report.
The investigators have a long history in working with GILS–related activities. Earlier studies included an analysis of locator–related legislation and policy instruments, a survey of existing or planned agency locators, and the design and specification for an agency–based, network–accessible government–wide information locator. Reports from these studies include:
The investigators brought this knowledge and previous experience with GILS to the current study.
1.1. SCOPE OF THE STUDY
"GILS" as a concept and a vision is broader than any single agency’s implementation. As discovered in the study, the term "GILS" means different things to different people. One can use the term to describe a number of things including a generic locator service, the technical specifications for a locator as defined in the GILS Application Profile (see National Institute of Standards and Technology, 1994), or specific implementations and systems providing locator services. The scope of the study became complex because of the range and number of agencies involved in the implementation, the differing views as to what GILS is and should be, and because aspects of GILS, for instance "improving public access," intersect with many other topics and initiatives.
The U.S. Federal implementation of GILS has been directed by policy statements, technical specifications, and implementation guidance. The scope of this evaluation was limited by design and intention to GILS implementations resulting from:
The investigators use the term "GILS" in the report, unless otherwise specified, with the following meaning:
U.S. Federal implementations of the GILS Application Profile according to specific policy instruments (OMB Bulletin No. 95–01), technical specifications (FIPS Pub. 192), and implementation guidance (NARA’s Guidelines for the Preparation of GILS Core Entries).
Terms such as "the Federal GILS initiative," "U.S. Federal GILS," "U.S. implementations of GILS," and "agency GILS," are synonymous with the meaning of "GILS" as defined above.
Any number of other jurisdictions and levels of government are involved in implementations of the GILS Profile. Initiatives at state and international levels often provide innovative approaches for consideration by the Federal government; no doubt this may be true of their GILS efforts. The investigators limited the study to GILS implementations directed by OMB 95–01; other Federal and non–Federal implementations of the GILS Profile were out of scope. Comparative studies of the U.S. Federal GILS and GILS initiatives of states or other national governments would be useful, and the investigators recommend the utility of such comparative studies in Chapter 5.
1.2. STRUCTURE OF THE REPORT
This final report offers findings from the evaluation study and recommendations developed by the investigators to improve the utility of the U.S. Federal GILS initiative. The findings are based on an analysis of the information gained through various study activities (see Chapter 3). The recommendations address policy, technology, implementation, and other changes to the U.S. Federal implementation of GILS. In addition, the investigators identify near–term and longer–term proposals to move GILS forward.
The chapters following this introduction include background on GILS, a policy and literature review, a description of study methodology, summary of findings and recommendations, discussion of priorities to consider when implementing the recommendations, and identification of areas for further research. Appendices to the report contain relevant GILS documents, detailed descriptions of the study activities, and summary results from various study activities. The appendices are a significant portion of the report and provide sources of data from which the investigators determined findings and formulated recommendations contained in the body of the report.
1.3. PRELUDE TO THE REPORT
The U.S. Federal government’s implementation of GILS has been an ambitious undertaking. Critics may point out limitations and flaws in the current coverage, implementation, and usability of GILS. Equally important, however, is recognizing the progress to date in developing a government information locator service and the commendable efforts by many people who have led and supported GILS implementations.
As a mechanism for users to discover, identify, select, and access government information, GILS faced and will continue to face many challenges, including satisfactory resolution of fundamental issues concerning information organization and access that the library and information profession has confronted for many years. In addition, GILS is implemented as a networked information service, and the arena of networked information discovery and retrieval (NIDR) is currently an active research area (e.g. the various digital library initiative). Operational solutions for many NIDR issues do not yet exist.
The Clinton Administration’s National Information Infrastructure: Agenda for Action intended GILS to be a "virtual card catalogue [sic] that will indicate the availability of government information in whatever form it takes" (Information Infrastructure Task Force, 1993). As happens too often, slogans can both enlighten as well as mask critical issues and challenges. In this case, the reference to the library catalog may obscure the complexity of that mechanism for connecting users with information. As shown later in this report, the complexity of implementing GILS as an agency–based, network–accessible "virtual card catalogue" was significant.
To place GILS development and implementation into perspective, one must recall the past century of library efforts in organizing and providing access to large collections of information. The late 19th century was a vital period for library theoreticians and practitioners who initiated the schemes for information organization and bibliographic control upon which present–day automated and online library information systems are founded. Over the past 100 years, librarians and other information professionals asked fundamental questions about how to connect users with relevant information, especially through the mechanism of the library catalog. They have tried to determine:
The answers to these and other information organization and access questions continue to occupy the attention of the library and information science profession. The library’s organization and access systems have evolved over the past 100 years because of theoretical and practical knowledge gained from implementing systems. The experience and lessons learned from efforts to connect users and information have informed each new generation of information organization and access mechanisms.
GILS designers and implementors are addressing long–standing issues related to the organization of and access to government information. Government information may have distinguishing characteristics. Many of the challenges of connecting users to government information, however, are similar to the challenges addressed by librarians and information professionals. GILS designers and implementors, however, have undertaken this initiative in a highly dynamic networked information and technology environment.
The developers of GILS recognized the need for standards to describe and represent government information resources (e.g., the GILS record data elements and structure) and the need for guidelines and rules for the creation of the records (e.g., NARA’s Guidelines). GILS developers also recognized the need to use evolving information technologies to store, search, and retrieve information (e.g., network technologies and information retrieval protocols). While these aspects of GILS design were—and continue to be—fundamentally appropriate, actual implementation experience can identify problems and raise questions as to the adequacy of even well–considered approaches. Two examples illustrate this point. The structured, standardized GILS records are an important contribution of the GILS initiative, yet in practice the records do not support currently stated goals of GILS for records management. A distributed, decentralized network-accessible locator service is architecturally elegant, but in the actual implementation, U.S. GILS is best characterized as a set of "agency information locators" that taken as a whole do not provide a consistent and coherent view of U.S. government information resources.
The U.S. Federal GILS experience is important in many respects, not the least of which is how agency GILS implementations are highlighting pre–existing conditions (e.g., agency information management practices or the lack thereof) and bringing new problems into finer resolution (e.g., the challenge of networked information discovery and retrieval, the importance of metadata and the challenges of its capture in a cost–efficient manner). The GILS experience also raises sensitive questions related to decentralization and centralization of information management authority, accountability, and responsibility in the digital age.
GILS was not intended—nor should it try—to provide a single solution to the information organization, access, and management problems of U.S. government information. In the world of information retrieval, many different information systems and services coexist, each with specific purposes and strengths in connecting users with the information they need. The same is true for government information. It is important, however, that GILS does the best job it can according to purposes appropriate for GILS. Defining and specifying what GILS is supposed to do is clearly needed. Most fundamentally, GILS is a "bibliographic instrument" for the networked information environment. GILS can assist users in discovering, identifying, selecting, and accessing U.S. government information. Since GILS is implemented as a networked information service, the early GILS implementation experience has highlighted important issues related to the specifications of bibliographic instruments when used to support networked information discovery and retrieval.
Patrick Wilson, an authoritative voice in the world of information organization, defines bibliographic instruments as having the primary function of listing and describing other writings. Through such instruments, users are able to identify, evaluate, select, and locate information that might be useful to them. In Two Kinds of Power: An Essay on Bibliographic Control (1968), Wilson identifies five basic specifications that must be clear—both to designers and users of bibliographic instruments—if they are to have "power" over information:
While Wilson was writing well before the advent of the networked environment, these five specifications are as pertinent to GILS as to any library catalog. In the coming pages, many of the identified problems with GILS implementations stem, in part, from the lack of understanding and appreciation for these five specifications. To its credit, GILS was an "early adopter" in the arena of networked information discovery and retrieval; concepts and approaches to networked retrieval have been under active development only in the recent years. As noted previously, this arena should be characterized as a research area since many of the problems of distributed search and retrieval have not been solved.
GILS can be seen as a first step in a new regime of the identification and organization of government information resources. If the past 2 years can be seen as an early implementation experiment in this regime, we will be able to look objectively and positively at the lessons learned, identify success factors, and look squarely at the shortcomings and failures. GILS has the potential to address long–standing government information organization and access issues, and the investigators intend this report to be of assistance in reaching that potential. Our recommendations for a refocusing of the GILS effort provides the next evolutionary step in the GILS initiative will continue the work done to date and build upon the experiences and lessons learned for improving public access to government information in the networked environment. We think it is essential, however, that policymakers draw a clear line of demarcation between the early GILS implementation period (i.e., 1995-1996) and a refocused GILS. One important aspect of such a demarcation is to acknowledge the lessons learned from the early implementation.
1.4. BENEFITS OF THE STUDY
Clearly, the chief product of any evaluation study is the findings and recommendations. In the study reported here, findings and recommendations constitute a major part of the report and will provide directions and strategies for a refocused GILS initiative. Beyond the use and importance of the findings and recommendations, a number of other benefits result from the evaluation study.
First, the study is a statement by the GILS Board, the Office of Management and Budget, and the sponsoring agencies which funded the evaluation that the U.S. Federal GILS implementation is important and deserves a careful review and assessment. Moreover, this statement recognizes that the GILS effort, if it is to develop and improve, needs an external evaluation to guide future decisions and action. In short, the commitment of resources and time to the GILS evaluation is in itself a declaration as to the overall importance of GILS. An easier step to have taken would have been not to conduct an evaluation.
Next, the process of the evaluation brought together a number of individuals with different perspectives on GILS to exchange information and learn from each other. During the evaluation effort, the investigators were very impressed with the level of interest and involvement in the study by a number of participants, as well as by others who were not study participants but very interested in the outcome of the evaluation. The process of the evaluation brought fresh attention to the GILS effort, raised its visibility within government, and provided a forum to discuss GILS and learn from various implementation experiences.
Third, as policy and evaluation research, the study developed and refined a number of important evaluation techniques that can be used by agencies and others to assess networked information services. Most important, we believe, are: the progress made on how to conduct Web server log analysis; use of online scripts for user assessments of networked information services such as GILS; and the techniques developed for the GILS record content analysis. Refinements to the methods of focus groups, site visits, surveys, and expert interviews also are important. Appendices to this report describe these techniques in detail.
The depiction of these efforts, reported largely in the appendices, may not do justice to the importance of the techniques undertaken here. Nonetheless, as the investigators discovered at a presentation to the GILS Special Interest Group meeting, April 23, 1997, there is considerable interest in these techniques and how to incorporate them as an ongoing part of GILS development and assessment. Participants at the meeting were very interested in applying these techniques to future GILS development at their agencies.
Agencies will need to develop formal measurement and evaluation techniques for their services and systems (such as GILS). This evaluation effort provides a number of useful guidelines and techniques for agencies developing performance and quality measurement techniques such as required by the Government Performance Results Act of 1993. The investigators believe that the assessment techniques and measurements used and tested in this study can be adopted or adapted by agencies for evaluating a variety of networked information systems and services.
Finally, the study provides a formal written assessment
of the U.S. Federal GILS effort after roughly 2 years of implementation.
As such, it provides a single source of information that all stakeholders
can review, discuss, and debate. Whether the evaluation results are taken
as benchmarks or beacons, the report provides a foundation for focusing
discussions and identifying the work ahead on the beneficiaries of GILS—its
users. Regardless of overall agreement with specific findings or recommendations,
the report provides a basis for all those interested in the future development
of GILS to begin the discussions and move forward with a refocused GILS
that will serve agencies and citizen users better.