APPENDIX C-1

APPENDIX C-1
SITE VISIT METHODOLOGY

Table of Contents

1.0. Introduction...............................................................................................................1
2.0. Site Visits as a Research Technique in Support of the Project Goals............................1
3.0. Selection of Site Visits................................................................................................1
4.0. Selection of Site Visit Participants...............................................................................2
5.0. Site Visit Demographics.............................................................................................2
6.0. Site Visit Data Collection and Data Analysis Activities................................................3
       6.1. Content Codes Used in the Database Management Software...............................3
              6.1.1. Content–Related Factors..........................................................................3
             6.1.2. Critical Success Factors............................................................................4
              6.1.3. Policy Related Factors..............................................................................5
              6.1.4. Standards.................................................................................................6
              6.1.5. Technology–Related Factors.....................................................................6
              6.1.6. Users–Related Factors..............................................................................6
7.0. Limitations And Conclusion........................................................................................7
        7.1. Limitations.........................................................................................................7
        7.2. Conclusion........................................................................................................7

Table C1-1 Site Visit Participants.....................................................................................2

1.0. INTRODUCTION

The investigators conducted one–day site visits to four agencies to solicit information about the GILS implementation process at each agency. Investigators held guided interviews with personnel from many administrative and functional areas. Site visits also included a focus group with agency staff, examination of relevant agency documentation, and tours/demonstrations of GILS sites.

2.0. SITE VISITS AS A RESEARCH TECHNIQUE IN SUPPORT OF THE PROJECT GOALS

Site visits provided the investigators with an opportunity for onsite interaction with agency staff involved with GILS implementations. Guided interviews with agency staff focused on individual roles as they related to GILS; on past, present, and future expectations with regard to GILS; on GILS policy issues; and on technology factors which guided agency implementation. Further areas that the investigators explored with agency staff included information about the agency’s selection of information resources for GILS record creation, and staff assessment of the data content within GILS records. Investigators probed for staff assessment of critical success factors and barriers encountered during the implementation process. Investigators paid particular attention to staff expectations about users of GILS, both from an internal agency perspective and from a public users perspective. This data collection enabled the investigators to gather information with respect to agency implementation experiences.

3.0. SELECTION OF SITE VISITS

In determining which agencies to visit, the investigators established preliminary criteria for agency site selection. In choosing sites, the investigators wanted a rich mix of the processes, people, programs, interaction, and structures under study (Marshall & Rossman, 1995). The preliminary criteria allowed categorization of agency by implementation effort:

Agencies implementing GILS servers
Agencies creating records but not implementing servers
Agencies which were not initiating any GILS activities (server implementations or record creation).

With these categories in mind, the investigators selected the Environmental Protection Agency (EPA), Government Printing Office (GPO), Department of Defense, Defense Technical Information Center (DTIC), and the Department of Treasury for site visits. Each of these agencies reflected different approaches and responsibilities toward GILS implementations:

EPA is an example of an agency which had the technical expertise to mount GILS records on its own server and had strong administrative commitment to GILS within the agency. Investigators visited this agency on October 23, 1996.
The Government Printing Office is an agency which brokers over 2,000 GILS records from 27 agencies. GPO staff created records for GPO resources needing GILS description. The remainder of the records brokered by GPO came from originating agencies that contracted with GPO to make their GILS records available. Investigators visited this agency on November 15, 1996.
DTIC (now part of Department of Defense’s Research and Engineering Directorate under the Under Secretary for Acquisition and Technology) hosts GILS records on its DefenseLINK site, making these records available both on the Internet and on DOD’s intranet. Prior to the formal emergence of GILS, DTIC had a need to create and maintain a GILS–like service. Investigators visited this agency on November 15, 1996.
The Department of Treasury is an example of an agency which mounted its records on GPO’s server and chose to create GILS records at the item level. Investigators visited this agency on January 10, 1997.

These four site visits captured different scenarios of agency GILS implementations and enabled the investigators to observe a variety of environments for GILS.

4.0. SELECTION OF SITE VISIT PARTICIPANTS

Site visits provided detailed understanding—from participants’ perspectives—of agencies GILS implementation activities and issues. The investigators interviewed as wide a variety of staff at each site as possible to ensure content–rich qualitative data from perspectives that would encompass the full range of an agency’s implementation experience. The investigators attempted to systematically identify agency staff for each site visit to provide an agency perspective on all five dimensions of the evaluation framework. Investigators identified the following categories as possible participants in GILS activities in each agency: Administrators/Managers, Policy Makers, Record Managers, IRM staff, GILS Record Creators, Librarians, and Agency Users. In arranging the site visit protocol for each agency, the investigators requested participation by individuals in each of these categories. Investigators also asked key agency contacts to suggest individuals who had been prominently involved in that agency’s implementation efforts. The demographics from the site visits reflect the actual group of people who participated in the agency’s site visit. The categories of staff interviewed during the site visit depended on that agency’s implementation scenario.

5.0. SITE VISIT DEMOGRAPHICS

The investigators talked to a total of 43 site visit participants. Of that number, 40 completed profile sheets. See Appendix D–1 for a copy of the profile sheet. Table C1-1 summarizes the types of staff who participated in the site visits.

Table C1–1
Site Visit Participants

	Treasury	DTIC	EPA	GPO
Total Participants	14	11	10	5

Categories of Staff
Program Analyst	3			1
Management Analyst	3			1
IRM staff	5	4	4
Librarian/Archivist	3		2
Technical Information Specialist		3
Administrator or Manager		1	1	1
Policy Analyst		1
Record Manager			2
Other		2	1	2

Average Yrs/Exp in present position	7	6.5	3.5	3.6

Nature of Work
CIO/Systems	6	3	2
Library	3	3	2
Record Manager	2	0	2
Project Manager	2	2	1	1
Other IRM	1	1	1
Public Information Officer		1	1
Other		1	1	4

6.0. SITE VISIT DATA COLLECTION AND DATA ANALYSIS ACTIVITIES

Participants interviewed during the site visit completed a profile sheet which included quantitative and qualitative questions related to GILS implementations. The profile asked respondents to assess, in a quantitative manner, familiarity with GILS policy documents. Agency staff used a Likert–type scale (from 1–5 in which 1 indicated Very Familiar and 5 indicated Not Familiar) to assess familiarity with GILS policy documents. The profile provided the investigators with assessments of agency staff expectations, training, and lessons learned. The profile sheet also included two questions which asked respondents to identity a favorite online alternative to GILS when trying to locate government information and the reason why this particular alternative was a favorite. These two questions enabled the investigators to identify online sources of government information which were used by agency staff and the personal preferences when agency users wanted to access Federal government information.

Prior to the site visit, the investigators developed interview questions, which were pre–tested by investigators and agency staff. These questions guided the interview process, though the investigators varied from the guided interviews when interviewee responses opened new avenues for data collection. Each site visit had facilitators and recorders who were investigators on the evaluation project. The facilitators and recorders at each session wrote a detailed description of comments made by participants, and an analysis of the issues discussed.

The investigators then created a database from these summaries and used database management software to manage the data collected. The investigators identified coding categories based on the actual data; the evaluation framework for the study sensitized investigators to categories of potential codes. The coding factors represented content found within the narrative summaries. The investigators used coding as a means of analyzing the data obtained from this data collection technique. Once analyzed, the coding scheme provided a data reduction technique for project investigators. As a result of this analysis, investigators were able to query the database for specific incidents of particular factors without losing the ability to focus on the data content from a holistic perspective. The codes used and their definitions are listed below.

6.1. Content Codes Used in the Database Management Software

The following list of codes and their definitions was used to analyze data collected from the site visits.

6.1.1. Content–Related Factors

The following codes identify content issues.

C–DC: Data content for GILS records

This code refers to decisionmaking with regard to data content of GILS records. It includes decisions about which fields to use, record display, granularity levels, and which systems/ product/privacy act systems to describe with GILS records. Issues connected with record creation activity or procedures use another code. References to NARA guidelines use this code if this was the source of the decision making.

C–IR: Effectiveness of Content for Information Retrieval (IR) Purposes

This code refers to the usefulness or value of data in GILS records for either agency staff or users. It includes reference to IR functionality and actual worth of using GILS for information purposes. Doubts or questions about the value of using GILS records to find specific information is included here. Concerns about data display are coded here. Issues about GILS–Lite are coded elsewhere.

C–MAIN: Content Maintenance

This code refers to the maintenance of the data content over time, not the original decisions when the records were created.

C–RC: Record Creation Activity

This code refers to the activity undertaken by agency staff to make GILS records. It includes information about which staff created the records, what procedures staff used, whether the agency made use of contractors to create GILS records, etc. Issues or procedures about the technology support (or lack of support) for record creation are coded elsewhere.

C–RM: Relationship between Record Content and Record Management Purposes

This code is used to bring together record content and record management issues. Policy issues about record management and GILS are coded elsewhere.

6.1.2. Critical Success Factors

The following codes identify critical success factors. Its use here is in the context of key or significant factors relating to agency implementation of GILS.

CSF–$: Financial Resources

If the agency identified financial resources as an issue or concern when implementing GILS this code is used. This code is used to indicate factors of a financial nature, both the presence or absence of funds.

CSF–AC: Agency Commitment

This code describes the attitudes or behavior exhibited by agency staff which demonstrated a commitment to the GILS initiative. A commitment to public access to government information uses another code.

CSF–AR: Agency Readiness

The key points to this code are pre–existing conditions which enabled the implementation of GILS to happen more effectively or in a more timely manner. Examples include preexisting use of locators, or metadata records. Factors such as technology, money, and administrative management are listed using separate codes.

CSF– B: Benefits

This code is used when there is specific compliance or links between an individual agency’s goals and GILS. For a broader commitment to GILS in general, the code P–B is used. Doubts as to the usefulness (lack of benefits) when implementing GILS are coded elsewhere but may serve to reflect the antithesis of this code.

CSF– BARR: Barriers

This code refers to barriers or problems identified by agency staff when implementing GILS. Long lists of barriers are coded with CSF–BARR; individual records may be coded with a second code if the barrier is in a specific area. By keeping the list of barriers within an agency together, a sense of the whole is maintained along with individual situation. Note: not all problems identified may be of a "critical" nature for an agency but are included here.

CSF–CU: Communication, Feedback with Users

This code refers to the agency’s communication with users during the implementation phase. Once the system became functional, ongoing communication is coded U–COMM. Examples of the CSF–CU might include focus groups.

CSF–MGMT: Effective Management Factors

The key point of this code is the presence or absence of effective managerial planning for GILS. This code would include a pre–existing organizational structure which aided the implementation of GILS or the use of skilled administrators to oversee the initiative. Generally the lack of effective management is coded as a barrier but if the issue includes both positive and negative aspects of management, this code is used.

CSF–SUS: Sustainability

The key point of this code is concerned with issues of sustaining the GILS initiative. Specific issues of record content maintenance used C–Main. This code treats broader issues beyond content that an agency must resolve to sustain its GILS initiative. It refers to policies, procedures, and activities involved in sustaining GILS whether implemented or suggested for implementation.

CSF–T: Time factors

This code highlights any concerns from agency staff which regard to time and implementing GILS. Frequently the need to implement GILS in a timely fashion was one of the factors that impacted the overall implementation.

CSF–TE: Technology

The key points connected with this code are the issues or questions about technology that impacted an agency’s implementation of GILS. This code is used to describe the effect that an agency’s technological infrastructure had on implementing GILS (e.g., whether or not to support its own server ). If the technology was a barrier, that code is preferred.

6.1.3. Policy Related Factors

The following codes identify policy issues.

P–95: OMB Bulletin 95–01 policy

This code includes questions or issues connected with the original OMB Bulletin 95–01 and policy issues directly addressed by it.

P–AC: Agency Culture

This code refers to pre–existing agency conditions which impacted the environment in which the GILS implementation occurred. This code provides insights into standard agency ways of doing things and the relationship between the culture within agencies and GILS.

P–B : Benefits of the Policy (awareness, belief in these benefits)

This code is used to describe general benefits that occur to users and agencies as a result of GILS. Specific benefits for an individual agency which motivated that agency to effectively implement GILS are coded as CSF–B. Doubts about the existence of benefits are coded elsewhere.

P–CAA: Communication Among Agencies

This code refers to communication which occurs across agencies. It refers to communication between agencies, not within a single agency. This code is preferred for cross–agency issues of a communication nature. Compare with P–IAG which is similar but not limited to communication.

P–CB: Costs/Benefits

The key words are costs/benefits and the questioning of the existence of benefits or value to GILS. If someone questions the value of GILS relative to the benefits, it is coded here. Specific barriers to implementation are coded CSF–BARR if they occur within a single agency. This code is not limited to a financial assessment of costs and benefits but rather gathers as many instances of references to benefits when coupled with costs.

P–CHAM: GILS Champion

This code is used to address the issue of the need for a GILS champion both within an agency and across agencies.

P–E: Evaluation

This code includes specific reference to the ongoing needs for evaluation or concrete evaluation measures agencies are using or are planning to use when evaluating GILS.

P–FOIA: FOIA

References to GILS and FOIA issues are coded here.

P–F/NS: Future Action or Next Steps in an Agency’s GILS program

The key to this code is the identification of policy and/or actions to implement in the immediate future. This code is reserved for lists of multiple suggestions written as a long list of an agency’s recommendations. Individual references might use C–Main, CSF–SUS, or other code. Possible examples: agency activities which are planned in the immediate future; recommendations about policy which are to be implemented by agency administrative staff. Some of the individual items on the list are coded separately by subject. This code preserves an agency–wide view.

P–GD: GILS Definitions

This code is used to describe how people or agency staff define GILS and its purposes. This code captures both articulated definitions and the general confusion about what GILS is supposed to be. General issues about what GILS is, is supposed to be, or confusion about either of the two are coded here.

P–GM: GILS Marketing

This code addresses the concept of GILS marketing and public relations. It is used when someone addressed the issue of marketing either within an agency or across agencies. General recommendations about the need to promote and publicize GILS are coded here.

P–IAG: Interagency GILS Factors

This code speaks to the relationship (or lack of ) between agencies or between GPO and other agencies. Any issue which crosses across agencies (except for communication) related to GILS is coded here.

P–IRM: IRM

Specific technical issues within an agency are coded as CSF–TE. This code refers to GILS policy issues specifically connected with IRM.

P–M/BC: Metadata and Bibliographic Control

This code speaks to the general concept of metadata and its use as a means that the public uses to find information. It includes the use of bibliographic control. Specific discussion of the needs for standards in the area of record content are coded S–RC.

P–PA: Public Access to Government Information

The key point connected with this code is the general issue of public access to government information.

P–RM: Record Management

This code is similar to C–RM but broader in that this category speaks to the general relationship between RM and GILS; issues concerned with specific fields or lack of data elements within records are coded as C–RM.

6.1.4. Standards

The following codes identify standards issues.

S–A/V: Appreciation and Value of Standards

The key point of this code is an agency staff’s awareness of, appreciation for, or value of standards. If the agency staff expressed an attitude that interoperability is a good thing or expressed an interest in promoting standards, this code was used.

S–DC: Standards for Record Content

Many GILS records do not follow uniform guidelines with regard to data content. The need for standards for record content is addressed here.

S–Z: Z39.50

The key word for this code is Z39.50. Specific reference to Z39.50 are coded here.

6.1.5. Technology–Related Factors

The following codes identify technology issues.

T–L: Links between Metadata and Full Text

This code refers to instances in which someone discuss links among GILS records, Web pages, and/or full–text documents. The discussion may include policy regarding these links, or desirability of having links where none exist, or expectations by users for links between text of documents, Web pages, and GILS records.

T–RC: Technology and Record Creation

This code refers to the issues of technology’s support of record creation.

T–T: Training

This code refers to instances in which someone discusses the need for training in use of GILS records or describes training activity undertaken by the agency in support of GILS.

T–W: The Web and Its Relationship to GILS

This code refers to the general relationship between GILS records and the Web. If users discuss the relationship between their use of GILS and the Web or if agency staff discuss practices or policies in the agency relating GILS activities and other Web based activities, this code is used. If the discussion emphasis is on links between/among GILS records, or full text documents, or Web pages, the code T–L is preferred.

6.1.6. Users–Related Factors

The following codes identify user factors.

U–AS: Agency staff

This code refers to agency staff as users of GILS records. If the agency staff are librarians, the code U–L is used. All other types of agency staff as users of GILS use this code.

U–COMM: Communication

The key point of this code is the ongoing channels which an agency establishes to communicate with users. Examples might include email links on GILS records or a user help desk.

U–L: Librarians

This code refers to librarians and their usage of GILS records.

U–P: Public

This code refers to the public as end users of GILS records and their experiences with GILS. If the information primarily refers to effectiveness of IR, it uses another code.

7.0. LIMITATIONS AND CONCLUSION

Site visits offered rich opportunities for data collection in that the investigators were able to evaluate GILS implementations in the context of each agency’s own territory. The investigators were able to learn first–hand about GILS implementations at each of the agencies from staff had recent direct experience with the purpose of the evaluation study.

7.1. Limitations

The investigators identified two limitations of the site visits as part of the evaluation process. One limitation involved in the use of site visits was the natural tendency by staff at each of the agencies to put an interpretation on events which would present the agency in as favorable a light as possible. A second limiting factor with this method was the dependence on the availability of particular staff for interviews on the site visit day. If key staff whose insight were considered important in understanding that agency’s implementation process were not available on the day of the site visit, the investigators contacted those individuals for subsequent interviews.

7.2. Conclusion

The use of site visits provided the investigators with access to groups of Federal agency staff who were directly involved with GILS. Having access to these individuals, enriched the data collection activity of the evaluation process by providing an extensive understanding of not only what actions were taken by staff but why these actions were taken. The occasion of the site visit also provided agency staff with an opportunity to reflect and analyze their collection actions with regard to GILS implementation.

Next Appendix
Previous Appendix
Table of Contents