1.0. Introduction..............................................................................................................1
       1.1. Organization of Material.....................................................................................1
2.0. Method Overview.....................................................................................................1
3.0. Data Summaries........................................................................................................2
       3.1. Participant Profile...............................................................................................2
       3.2. GILS Content Expectations................................................................................3
       3.3. GILS Service Expectations.................................................................................4
       3.4. GILS Record Characteristics..............................................................................6
       3.5. GILS Information Space and Ownership.............................................................6
       3.6. GILS Nomenclature...........................................................................................7
       3.7. Searching Reflexes and Relevance Judgments......................................................8
4.0. Findings, Discussion, and Recommendations..............................................................8
       4.1. Participant Profile Script Items............................................................................9
       4.2. Objectives-Driven Script Items...........................................................................12
5.0. Summary of Recommendations..................................................................................35
       5.1. Increase Users’ Searching Confidence................................................................35
       5.2. Improve GILS Niche in Information Space.........................................................36
       5.3. Improve GILS Efficacy in NIDR and Revise NARA Guidelines Accordingly.....36
       5.4. Improve the Quality and Consistency of GILS Records......................................37
6.0. Areas for Further Research.......................................................................................37
7.0. Conclusion...............................................................................................................38 

Attachment E3-1: GILS Record from Script Item S28.....................................................39

1.0. INTRODUCTION

To capture user perceptions about and reactions to GILS concepts and implementations, the evaluation featured an exploratory technique based on a set of scripted service encounters. The scripted online user assessment achieved four goals. First, the online sessions permitted in-process, "front-line," collection of data concerning user assessments of GILS—as opposed to "recollection" of assessments after GILS use. Second, it elicited highly qualitative responses to a concept (i.e., rather than the more traditional aims of user assessments such as quantification of relevant "hits" or usage patterns). Third, the findings provide a degree of insight into the cognitive processes of users in the online, networked environment. Last, documentation of lessons learned during development and deployment of the new exploratory technique (see Appendix C-5 Scripted Online User Assessment Methodology) provides a basis future researchers to adapt the script and delivery techniques to their specific environments and objectives. Data such as those discussed in the following sections are crucial to understanding user perceptions, expectations, and behavior during networked information discovery and retrieval (NIDR), and in advancing the quality of GILS accordingly. 

1.1. Organization of Material 

Section 3.0 Data Summaries aggregates significant results in terms of the user session objectives. Detailed results of the sessions in Section 4.0 Findings, Discussion, and Recommendations are presented in order of appearance on the script and are prefaced by an alphanumeric code designating whether the data collection item (script question) was designed to profile the user (P) or support the specific objectives of the session (S). Each item in Section 3.0 Data Summaries tables cites the corresponding S or P number as found in Section 4.0 Findings, Discussion, and Recommendations. The codes "UNT" and SU" in the Section 3.0 data summary tables indicate that the finding resulted from the post-session debriefing at University of North Texas (UNT) or the post-session focus group at Syracuse University (SU). 

Sections 5.0 and 6.0 present a summary of recommendations and opportunities for further research, respectively. 

2.0. METHOD OVERVIEW 

Graduate and undergraduate student "users" unfamiliar with GILS were oriented to the nature and purpose of their participation by means a 5-minute verbal introduction by the investigators. They were subsequently asked to record answers to more than 50 multiple-choice, free-form expression, and true/false questions as they navigated "real life," "real time" Government Printing Office (GPO) GPO Access and Environmental Protection Agency (EPA) GILS systems according to a scripted set of encounters. The script was based on results of the record content analysis (see Appendix E-2 Record Content Analysis Findings, Discussion, and Recommendations) and investigators’ ongoing search/retrieval experience with various GILS. The questions were designed to elicit user feedback concerning GILS content and service expectations, record design, orientation in information space, adaptation to the metadata construct (e.g., searching reflexes), and, perhaps most importantly, users’ assumptions about GILS—all on the basis of this 1-hour first-exposure to scripted transactions. In addition, investigators conducted debriefing sessions where users were informed generally of GILS scope and purpose and asked to elaborate on intellectual and emotional impressions created by the scripted. The qualitative data from the sessions were entered into a database to facilitate disclosure of patterns related to users’ reactions to GILS as a service concept and to GILS product (search options, results set, and records). As with the record content analysis, investigators recorded suggested improvements to the development and execution techniques for scripted online-user assessment in order to optimize recommendations to agencies interested in adopting the techniques. 

The scripted online user assessment script was developed and the sessions conducted during February 1997. Thus the results presented do no reflect any subsequent modifications to GPO’s and EPA’s GILS system configurations, capabilities, and user interfaces. 

3.0. DATA SUMMARIES 

The following tables summarize significant findings in terms of objectives for the user sessions. Investigators strongly recommend that interpretation of the following findings be guided by the complete data and discussions provided in Section 4.0 Findings, Discussion, and Recommendations, which are organized in script order, by an alphanumeric code designating whether the data collection item (script question) was designed to profile the user (P) or support the specific objectives of the session (S). Each Section 3.0 data summary table cites the corresponding S or P number as found in Section 4.0 Findings, Discussion, and Recommendations. The codes "UNT" and SU" in the Section 3.0 data summary tables indicate that the finding resulted from the post-session debriefing at University of North Texas (UNT) or the post-session focus group at Syracuse University (SU). 

The scripted online user assessment script was developed and the sessions conducted during February 1997. Thus the results presented do no reflect any subsequent modifications to GPO’s and EPA’s GILS system configurations, capabilities, and user interfaces. 

3.1. Participant Profile 

The 10 participants represented reasonably capable but "GILS unaware" users of online networked information resources. The following table summarizes pertinent data. 

*These (S) items, although appearing in the actual script portion of the instrument, reflect information about the user profile and thus are summarized here. 

3.2. GILS Content Expectations 

The following table summarizes session participants’ expectations for GILS content—full-text of documents vs. metadata, subject matter and resource types, quality, scope and extent of collection, and record and resource aggregation ("distance" from satisfaction of an information need). 

3.3. GILS Service Expectations 

This table summarizes findings concerning session participants’ expectations for GILS serviceability in NIDR. It includes their reactions to predictability of results, fielded searching, system errors and response time, hyperlinks, and GPO-EPA system consistency. 

 3.4. GILS Record Characteristics

Study participants’ reactions to characteristics of GILS records—e.g., cosmetic appearance, record length, element display order, and formatting are summarized below. 

3.5. GILS Information Space and Ownership

The following table presents summary findings about study participants’ perceptions of GILS navigability, centralization concepts (loci of services and products) and implementation architecture (network distribution), and availability and authority of resources. 

3.6. GILS Nomenclature 

The following table summarizes participants’ reactions to GILS presentation and use of bibliographical and NIDR terminology. 

3.7. Searching Reflexes and Relevance Judgments 

Participants’ preferences for full record vs. fielded searching, their relevance improvement tactics, and perceptions of user sophistication requirements (education and training) are summarized in the following table.

4.0. FINDINGS, DISCUSSION, AND RECOMMENDATIONS

Detailed results of the sessions are presented below, in tabular form as appropriate, in order of the data collection item’s appearance on the script. The prefatory alphanumeric code designates whether the data collection item (script question) was designed to profile the user (P) or support the specific objectives of the session (S). Throughout this section, the discussion of findings is correlated to results of the record content analysis (see Appendix E-2 Record Content Analysis Findings, Discussion, and Recommendations) as appropriate. In addition, recommendations based on interpretation of the findings are provided for improving GILS from a user perspective and for future research to clarify issues. 

The scripted online user assessment script was developed and the sessions conducted during February 1997. Thus the results presented do no reflect any subsequent modifications to GPO’s and EPA’s GILS system configurations, capabilities, and user interfaces. 

4.1. Participant Profile Script Items 

The first 10 items plus 3 later in the script (S13c, S323, and S32k) captured demographic and other information about the participants—such as status (e.g., student, private citizen, etc.) and government-information search frequency, methods, resource types, and knowledge/attitudes about government information—providing a context for evaluating expectations and responses. Results of these items are summarized in the following section. 

P1. How do you chiefly acquire or refine you online searching skills? (Circle one.)
Findings: 8 of 10 participants reported "self teaching through trial and error"; 1 participant reported "professional training" and 1 reported "reading online Help manuals." No one reported "applying knowledge of database design."
Discussion: Users, even library students, do not appear to rely on professional training in database design or searching methods for NIDR. Users may be reluctant to consult online help manuals to avoid interruptions in search-thought processes or having to interpret overly complex or technical-jargon-laden instructions. Also, online help might not be available in their experience, or, they may enjoy the challenge of "cracking" the system.
Recommendations: Present concise, comprehensible search instructions on the same page as the search input mechanism. Provide an example.

P2a. How often do you search print sources of government information?
Findings: 1 of 10 participants searches daily; one-third search monthly. The remainder search as required for academic credit and once or twice per year.
Discussion: Participants were not "power users" of printed government information. Adoption of access mechanisms for print sources to networked information resources may not bear fruit.
Recommendations: Additional research may be warranted to determine user satisfaction with agency name as the primary access point in traditional sources. 

P2b. How often do you search online sources of government information?
Findings: 2 of 10 participants reported weekly searching and 2 reported monthly. The remainder search online as required for academic credits and up to 4 times per year.
Discussion: Overall, participants search online sources more frequently than print sources, although no one reported a frequency that in the investigators’ judgment is required to gain retrieval proficiency.
Recommendations: Design GILS systems to accommodate the infrequent searcher. 

P3a., P3b. Have you ever read or heard about the U.S. Government Information Locator Service (GILS), or actually used it?
Findings: Only 1 participant was aware of GILS: "found GILS when searching for class requirement."
Discussion: A small sample of users with backgrounds that one might expect to include a GILS encounter (see S5) were unaware of the service.
Recommendations: Advertise in government publications and libraries; incorporate GILS linkage into agency and White House homepages; register/index GILS homepages with popular Web search engines. 

P4. Approximately how long have you used the Internet?
Findings: The maximum length of Internet usage was 4 years (1 participant); the minimum was 2 months (1 participant). On the average, users reported slightly more than 2 years’ usage.
Discussion: Assumptions about potential GILS users in the academic environment (see S5) may reasonably include a relatively long Internet exposure/experience period.
Recommendations: Additional research is recommended to elucidate relationships between longevity of experience, self-ratings of "familiarity" or "proficiency," and satisfaction with system results. 

P5. Please circle the letter that most closely matches your current status
Findings: Participants in the user study included 1 political science undergraduate student, 6 library science graduate students, 1 art undergraduate student, 1 history undergraduate student, and 1 "private citizen."
Discussion: The participants represented a reasonably "GILS-capable" population in terms of education level and subject orientation.
Recommendations: Additional user assessments should involve corporate librarians, small business owners, school teachers, political action group members (e.g., League of Women Voters), etc. It is suggested that Public Information and Freedom of Information Act Officers develop profiles of "print" information seekers, and that webmasters do likewise for agency website visitors, to optimize sampling for GILS user assessments.

P6. What types of government information do you seek most frequently? (Circle up to three).
Findings:

Discussion: The seeking of information on government activities and for public notices may reflect a "news"-consumption behavior (i.e., a desire for "awareness" as opposed to specific, targeted information retrieval for "question answering") among students.
Recommendations: Additional research is recommended along the lines of "what was on your mind the last time you recall deliberately searching for or monitoring government information." For example, users seeking to satisfy a specific and direct but occasional information need may prefer the approach planned by USPS’s WINGS (Web Interactive Network of Government Services) <http://www.wings.usps.gov/Topten/>, which will present information on, among other things, tax-return filing, requesting birth certificates, and job searching. 

P7. Please circle letter(s) matching your experience seeking government information, in print or online
Findings:

Discussion: The experience of browsing was shared by the large majority of participants, as opposed to reliance on either a bibliographic tool (catalog or index) or human intermediary. No one reported experiencing a lack of user’s instructions, which may indicate that they do not seek them out or that they find them adequate (see S1).
Recommendations: Additional research may find a relationship between "character of experience" and "type of government information sought" (see S6).

P8. If you were to enter search terms into an online "information locator service," what would you expect in return? (Circle all that apply).
Findings:

Discussion: The expectations followed by an asterisk in the above table most closely match GILS. Bearing in mind that only 1 user had "heard or read about…or actually used GILS," it appears as though the participants had, prior to exposure, a fair idea of what to expect from the service. The notable exception is almost half of participants believing that GILS might provide full text of documents. It is interesting to note, however, that 3 of the 4 checking the "full-text" option also checked the "statements about where relevant information is stored and how to obtain it" option. Of the nine options presented in the script, the maximum N for any particular option was six and the minimum was one ("names of experts in the subject"). These data may reflect some users’ notions about the varieties of product returned by searchable online resources and/or uncertainty about the terms "information" and "locator." Conversely, 1 participant’s response indicated no ambivalence about service expectations: "exactly what I want."
Recommendations: Results of this and other items (see P9 and P260) support a recommendation for a more clear communication of GILS purpose and approach. The reader is referred, for example, to FedWorld’s GILS site <http://www.fedworld.gov/gils>, which states directly above the search-form input boxes: "Please also note that GILS records are intended to allow you to learn about what government information is available, not to be [FedWorld’s emphasis] the information that you might be seeking! [FedWorld’s punctuation]." It is further recommended that agencies’ avoid linking to or quoting verbatim policy documentation for the purpose of introducing users to GILS functionality.

4.2. Objectives-Driven Script Items 

This section presents results of script items designed to support the research objectives—i.e., to capture user perceptions about and reactions to GILS concepts and implementations. (Note: The script was developed and the sessions conducted during February 1997. Thus the results presented do no reflect any subsequent modifications to GPO’s and EPA’s GILS system configurations, capabilities, and user interfaces.) 

S1. There are many options listed underneath Individual Agency GILS databases on GPO Access. What do you think these might represent? (Circle all that apply).
Findings:

Discussion: While it is difficult to interpret this finding in isolation from the efficacy of GPO Access GILS search page user-interface design, responses may indicate uncertainty about government information creation and dissemination policy and its implementation. Users with a priori knowledge of GPO’s "distribution" mission may be especially confused; no explanation of the comparatively low incidence of "information creators" is offered. This issue is informed by S30a and S30e responses. The "Other" response also indicates some doubt or confusion; "various" is nonspecific, and the participant seems to imply that there is one GILS database to which agencies contribute.
Recommendations: GPO should include a straightforward statement on the GILS search page(s) to the effect that the databases contain an agency’s GILS records of information resources created and available from that agency and that GPO GILS listing may be incomplete. In addition, the name "GILS" should be re-thought because it implies the singular. It may not be reasonable for a user to intuit that GPO’s GILS is not the same as GPO Access’ GILS; the former concept is comprehensible but the latter might be named more aptly "A Collection of Agency GILS." Likewise, since FedWorld’s "GILS" doesn’t exist in the singular it should not be so labeled on the website. All nonbrokered agencies’ sites should be entitled "[Agency/Bureau/Etc. Name] GILS" rather than "GILS." S25a,b,c and S30e and S30a also address this issue. 

S2. Would you assume that [GPO Access GILS] "mandatory GILS core elements" means that these fields always contain data?
Findings: 6 of 10 participants answered this question affirmatively.
Discussion: The term "mandatory" is misleading; its common synonym is "obligatory." Users familiar with commercial online search services, wherein "field" availability/existence is specified, may be especially prone to confusion.
Recommendations: The results of GILS record content analysis (see Appendix E-2 Record Content Analysis Findings, Discussion, and Recommendations) indicated an inconsequentially low incidence of "mandatory" element data population, even among those records designated as core by the presence of "US Federal GILS" in the local subject term element. This, coupled with the ambiguity of the words "mandatory" and "core," and perhaps even "element" (rather than "field"), should prompt an examination of the utility of exposing the user to the concept at all. S32m also speaks to this issue. 

S3. Please rate how comfortable you would feel using the [GPO Access GILS] options presented under "Select one or more of the following fields…to search"
Findings: Prior to any searching and with instructions not to click on the hypertext to receive a definition, participants were most comfortable with Purpose, Title, Cross Reference, Date of Last Modification, and Sources of Data. They were least comfortable with, or most unsure of using, Original Control Identifier, Spatial Reference, Schedule Number, and Control Identifier.

*1 participant did not address this option.
Discussion: Of common bibliographic metadata, participant confidence in the terms "Title," "Cross-Reference," and "Local Subject Index" was not unexpected. "Date of Last Modification" was revealed as problematic in the GILS record content analysis (see Appendix E-2 Record Content Analysis Findings, Discussion, and Recommendations) vis-a-vis misunderstanding that this element referred to the information resource being described rather than the record itself; the data above and that from S29 may corroborate this finding. The incidence (5) of "certain" understanding of or comfort using "Abstract" may be low, especially for the academically-oriented study group, and may reflect a distrust of the concept due to misappropriation of the term by many popular Web search engines.
Recommendations: Transaction log analysis may provide additional insight into users’ choice of elements for fielded searching. 

S4. [Participants were asked to read the GPO Access GILS-supplied definition of a field name they marked "unsure" of in S3] Does this definition affect your confidence in using this field for searching?
Findings: 7 of 10 participants noted that the selected definition "increased" confidence; 2 reported "no change" in confidence; and 1 reported a "decrease" in confidence.
Discussion: Nearly one-third of participants did not find the GILS element definitions helpful. S5, S14, S24, and S27 results also address fielded searching.
Recommendations: Subject the definitions to reading-level (e.g., Flesch Reading Ease, Flesch-Kincaid Grade Level, etc.) and grammar checks to compute "fog index" by identification of incomplete clauses, jargon, passive voice, characters/syllables per word and words per sentence. These checkers are standard on most (fully-installed) popular word processing programs. Provide an example of the utility of the field to increase retrieval precision. 

S5. Please check [of GPO Access GILS-supplied fielded searching options] below which ONE field you would select for locating information about native americans.
Findings: 4 of 10 participants selected Local Subject Index and 2 selected Controlled Vocabulary. The remaining 4 selected Title, Cross Reference, Agency Program, and Control Identifier.
Discussion: The reliance on controlled vocabulary for subject-driven searching as expressed by more than half of the participants may be a function of background or inclination; 3 of these 6 were library science students. There may also have been an assumption that these fields are always populated (see S2). The choices of Title and Agency Program may be interpreted as bids for high retrieval precision. However, the remaining choices appear unsupported by the current analysis, and may further indicate nomenclature problems (see S4).
Recommendations: In online help, provide an example of the utility of each field to increase retrieval precision. 

S6. [As a result of executing a query by using All GILS Records on GPO Access GILS Site, the field selected in S5, and an (unrecorded) term "relevant to the concept of native americans"] How many total hits did you receive?
Findings: Of the 9 participants executing this search, the maximum report was 40 hits and the minimum was 2. Participants averaged 17 hits.
Discussion: The 2 participants receiving (default maximum) 40 hits searched the "Agency Program" and "Controlled Vocabulary" elements, respectively. 2 of the 3 participants receiving 2 hits (default failed-search Query Report and Database Catalog) searched the "Local Subject Index" element and the other searched "Control Identifier." A search on the "Cross Reference" element produced 32, "Title" produced 3, another’s "Local Subject Index" search produced 4, and another’s "Controlled Vocabulary" search produced 31 hits. Use of controlled vocabulary may have increased recall.
Recommendations: Survey the agencies using controlled vocabulary and determine, through log transaction analysis, whether the practice increases retrievals. 

S7. [As a result of executing a query by using All GILS Records on GPO Access GILS Site, the field selected in S5, and an (unrecorded) term "relevant to the concept of native americans"] What is your reaction to the number of hits?
Findings: 2 participants did not answer this question.

Discussion: Unfortunately, the participant expressing "pleasant surprise" received only 2 hits—the default "failed-search" results. Those "not surprised" received 40, 40, 4, and 2 hits, respectively, which may be indicative of a range of search-skill confidence levels. The "disappointed" participant received 3 hits, and the "overwhelmed" one received 32. It is interesting to note that no one expressed a desire to reformulate the query to produce fewer or greater hits. And, no one, with this first query in the script, was frustrated to the point of abandoning GILS.
Recommendation: With a more diverse and larger user sample, attempt to reproduce these results and correlate them with published findings of end-user search recall satisfaction (Note: no relevance judgment was required on this exercise). 

S8. If you were to select one or more [GPO Access GILS-supplied] hits for closer examination, which factor would most likely influence your selection?
Findings: 1 participant did not answer this question.

Discussion: The preference for "appearance of search terms in the title" may reflect a lack of understanding or value of WAIS relevancy ranking: "Relevance is computed based on several factors, including the occurrence of the search terms in the document title, the frequency of the terms as a percentage of the total document size and conformance with the exact search phrasing."
(Ref: <http://www.access.gpo.gov/su_docs/aces/aces180.shtml?desc014.html#cont09>.
While WAIS will return highest "score" first, investigators included the "order of appearance" choice to acknowledge that several records may carry the same "score." It is interesting to note that no one assigned importance to file format (HTML, TEXT, PDF, SGML, etc.) or size (expressed in bytes).
Recommendation: None. 

S9. Please characterize and explain your single FIRST REACTION to this record [retrieved on the basis of seeming "most promising" from the results list of the query on native americans at GPO Access’ GILS]:
Findings: Some participants noted multiple "first" reactions.

Discussion: This point in the script represented the first time 9 of 10 participants looked at a GILS record (see P3a). 7 reactions included expressions of "disappointment," and a combined 4 reactions were frustration or confusion. The most frequent comment concerned the lack of full text (4 participants); 2 expressing disappointment and 2 expressing surprise. (Both participants expressing "disappointment" over the lack of full-text also indicated in P8 that an online "information locator" might contain full text, but those expressing "surprise" did not!) 4 participants were disappointed by a perceived lack of relevance, and 2 participants were disappointed by their respective records’ cosmetic appearance. Positive comments included "surprise" that a record appeared and "interest" evoked by rapid system response and availability of a resource at no cost. The comments of 2 users "frustrated" with their records are self-explanatory.
Recommendations: While the investigators believe that the options "surprised" and "interested" provided ample opportunity for positive feedback about participants’ first impressions of a GILS record, it is recommended that a more direct record of user reactions be captured through "talk-aloud" protocol or a completely open-ended question. 

S10. Please describe anything you consider to be peculiar or in error [about or in the retrieved record]:
Findings: Participants reported the following about their respective records (note that the script did not control which record users examined):

• Some of the descriptions do not appear to match their category (i.e., under agency program it describes how commission programs are designed to increase understanding—no actual info on agency program)
• Abstract should be abstract of the info, not only [just] description as to "report" and page nos.
• "Onondaga Nation" should retrieve hits with BOTH terms

Discussion: 2 of 4 participants responses to this item concerned unfulfilled expectations for content (in Agency Program and Abstract elements) and one concerned retrieval logic (see also S9). The record content analysis (see Appendix E-2 Record Content Analysis Findings, Discussion, and Recommendations) may corroborate incidence of element misuse as perceived by the study participants. An additional comment: "Five out of 12 hits were clicked and no data was returned. Could be time of day (12 pm)" was a result of poor wording of the question and was not considered in the analysis.
Recommendation: Agencies are encouraged to enlist objective content reviewers to evaluate conformance of fields’ content to qualitative descriptions and examples provided The government information locator service: Guidelines for the Preparation of GILS Core Entries (National Archives and Record Administration, 1995a). In addition, search engine or search/retrieval-standard performance should be evaluated against system documentation
(e.g., for GPO Access GILS [WAIS]:
<http://www.access.gpo.gov/su_docs/aces/aces180.shtml?desc014.html#cont01>). 

S11. What do you think would happen if you were to click on this record’s [retrieved deliberately by script instructions] hypertext title?
Findings: Note that the title of this record is "[Agency name] Library Services." 2 participants did not answer this question.

Discussion: More than half of the responses correctly assumed a link to the agency’s website. The remainder chose possibly plausible options.
Recommendation: In this particular instance, hypertext tagging of only the agency’s name within the title of the record (i.e., Agency Name Library Services) might have resulted in a higher rate of correct answers. In all cases of hyperlinkage in GILS records, it is recommended that an objective party review the context of the link to ensure that the record creator is not over-assuming an intellectual "hop" that a user might not be prepared to make. The record content analysis (see Appendix E-2 Record Content Analysis Findings, Discussion, and Recommendations) also addresses this issue. 

S12. Please rate your feeling about the length of this record [retrieved deliberately by script instructions] relative to your satisfaction:
Findings: Of the 8 participants responding to this item, 7 selected "Just right; it presented the necessary information" and 1 selected "Too long; it provided more than I needed to know. No one selected "Too short; it doesn’t present enough detail."
Discussion: The subject record contained 14 elements and 4473 bytes—a relatively "short" record according to the record content analysis (see Appendix E-2 Record Content Analysis Findings, Discussion, and Recommendations). These limited data may indicate that users prefer to err on the side of brevity.
Recommendation: Additional user-based research could examine which elements contain the information (or information pieces) sought and/or which elements are consistently ignored by users. 

S13a. Do you think the search term "trade agreements" would have produced a "hit" on this record [entitled "[Agency Name] Library Services" and retrieved deliberately by script instructions]?
Findings: Of 8 participants responding to this question, 6 correctly answered in the affirmative and 2 responded "no."
Discussion: The term "trade agreements" was one item in a bulletized list of subjects embedded in the record’s Abstract element. This question was included to gauge participants’ recognition or acknowledgment of search mechanics—i.e., in an indirect fashion determine whether, if a user were to search on "trade agreements" and receive a hit entitled "[Agency Name] Library Services" he or she would proceed to retrieve the record for further examination. In actuality, the subject database contains only 11 GILS records, and a search on "trade agreements" produces 2 hits (neither of which feature the term in the title; see S8), the second being "[Agency Name] Library Services." While it may be reasonable to assume that an agency library contains information on a wide variety of subjects, it may not be realistic to expect a user untrained in searching or unfamiliar with agency libraries to recognize this; the high recognition rate among the participants may be due to study group demographics (see P5).
Recommendation: Especially where records itemize subject areas covered, as in this case, the terms might be placed more appropriately in the Local Subject Index element. The GILS record content analysis (see Appendix E-2 Record Content Analysis Findings, Discussion, and Recommendations) discusses the issue of information resource granularity more thoroughly. 

S13b. If you were looking for information about the [Agency Name] library, would this record [entitled "[Agency Name] Library Services"] satisfy you?
Findings: 7 of 8 participants were satisfied by this record.
Discussion: See S8 concerning users’ evaluative criteria and S16 (relative quality rating).
Recommendations: "Model" records, as determined through user-satisfaction studies, should be readily available to serve as an example to record creators and as a benchmark for evaluators. 

S13c. [Relative to record entitled "[Agency Name] Library Services"] Did you know that Federal agencies have libraries?
Findings: 8 participants answered this question; 6 with "yes"; 2 with "no."
Discussion: This question was included to bring perspective to S13d. Given the preponderance of library school students in our user sample, no conclusions are offered.
Recommendations: See S13d.

S13d. [Relative to record entitled "[Agency Name] Library Services"] Would you have expected to find a Government Information Locator Service record that describes a library?
Findings: 8 participants answered this question; more than half (5) answered affirmatively; 3 said "no."
Discussion: Interestingly, of the 5 "yes" respondents, 3 had reported "not expecting" an online information locator to provide "statements about where relevant information is stored and how to obtain it" (S8c).
Recommendation: Agencies with public service "traditional" libraries should cross-reference that resource within applicable records. The GILS record content analysis discusses the issue of information resource granularity more thoroughly (see Appendix E-2 Record Content Analysis Findings, Discussion, and Recommendations). 

S14. [Relative to a scripted retrieval of two duplicate records from a selected agency at GPO Access’ GILS] Why do you think two apparently identical results have been returned? (Circle all reasonable possibilities)
Findings:

Discussion: Note that the users were viewing a search-result page only at this point, not records themselves. The high incidence (7) of belief in two record sources may support a misunderstanding of database ownership and/or placement in information space (see also S1 S30a, and S30e) because participants executed this search in the "selected agency" mode rather than "all records on GPO Access GILS." The notion that titles had been truncated for display is certainly reasonable. It is interesting that only 2 of 10 users considered the possibility of system failure and that only 2 users attributed cause to human error (the searcher and the record creator, respectively). This result points to a tendency among participants to consider that "something about this thing called GILS" is at fault.
Recommendation: A mechanism such as input masks or prevent-duplicates indexing feature should be implemented by all GILS providers. The presence of duplicate records might easily erode user confidence in the quality of records or management of the system and can be easily avoided. 

S15. Please describe anything you consider to be peculiar or in error [about this record retrieved according to the script]:
Findings: 8 of 10 participants responded to this question, and all but 1 of these noted the formatting error (apparently undelimited ASCII text; no hard returns):

• Wrap-around disabled; page info incomplete (avail)
• No HTML, only text was available
• No text wrapping—people like to scroll up and down, not left to right
• Page width is disconcerting and what kind of public personnel management is it. No additional link.
• Having to keep scrolling to the right to read entire line
• That you have to scroll right and left to read is a problem
• Text runs horizontally only—must scroll to right to read text
• Have to expand screen to read full record, not enough information, poor information provision

In addition, 1 participant noted that the record "ended" prematurely (due to the formatting error) and 2 participants complained about the quality or incompleteness of the information itself. The comment concerning "no HTML" being available was assumed to be the result of a scripting error and was not considered in the analysis.
Discussion: Participants recognize formatting errors.
Recommendations: Record creators and/or approvers should view product as displayed by a browser selected on the basis of published usage reports (e.g., "Browser Battle" July 1996 Internet World p. 40) or their agency website access log analysis. 

S16. Of the two agencies’ records [scripted for retrieval] describing libraries, which is best?
Findings: Of 8 participants answering this question, 2 expressed "no preference"; the remainder thought the [agency name] library record was "better."
Discussion: The inadvertent omission of a question concerning "peculiarities and errors" vis-a-vis the preferred record precludes a definitive interpretation of this finding. However, it should be noted that the preferred record did contain hypertext, was correctly formatted and available in HTML, and contained discernable elements; the nonpreferred record did not feature these characteristics.
Recommendations: "Model" records, as determined through user-satisfaction studies, should be readily available to serve as an example to record creators and as a benchmark for evaluators. 

S17. What characteristics distinguish the two records [describing agencies’ libraries and scripted for retrieval] in your mind?
Findings:

* 2 participants did not address this option; 1 participant wrote "ignorable."
Discussion: These results are somewhat inconclusive absent operational definitions of the characteristics (e.g., "format" could be "accurate" or provide "hotlinks"), but may support the finding suggested in S9 that users place value on records’ cosmetic appearance. It is interesting to note that 5 participants’ responses imply that one or the other record was relatively "incomplete" when only 2 noted this in S15. The characteristics of "consistency" is further addressed by S32f, m, and n. In addition, it is noted that only one of the two records featured a hotlink and users were instructed (in the interest of time) not to pursue it; despite this limitation 2 participants recalled this as a distinguishing characteristic.
Recommendation: "Model" records, as determined through user-satisfaction studies, should be readily available to serve as an example to record creators and as a benchmark for evaluators.

S18a. Describe how you would use [GPO Access’] GILS to find the toll-free number for ordering a statement of earned social security benefits. What would you choose in the Make your [database] selection(s) scrollbox?
Findings: 9 of 10 participants completed this question. Of the 9, 8 reported that they would select "Social Security Administration," the ninth would select "all records on GPO Access site."
Discussion: (Note: the script was prepared and sessions conducted prior to the recent controversy surrounding availability of service via the agency’s website.) This finding indicates that nearly all participants recognized the availability of a relevant and agency-specific database; the "all records" respondent may have been motivated by recall or perceived search efficiency rather than precision.
Recommendations: None. 

S18b. Describe how you would use [GPO Access’] GILS to find the toll-free number for ordering a statement of earned social security benefits. Would you use "fielded search?"
Findings: Of the 9 participants answering the question, 3 would use fielded searching, the remainder would not.
Discussion: At this point in the script, participants had executed only one fielded search (S7-S10), for which the majority of reactions were negative (see S9). This initial turn-off may account for an apparent reluctance to use fielded searching even for this relatively specific information need.
Recommendation: It is recommended that GILS designers perform a confirmatory analysis that fielded searching improves retrieval precision in GILS for both known-item and exploratory searching.

S18c. Describe how you would use [GPO Access’] GILS to find the toll-free number for ordering a statement of earned social security benefits. What would you type into the "Enter your search term(s)" textbox?
Findings: 9 of 10 participants answered this question, providing the following search strings (presented here with the participant’s method of acquiring online searching skills from S1):

Discussion: The script instructed participants to read a brief and accurate description of boolean operators in preparation for a previous searching event. Comparable information was present on their screen during the present exercise (<http://www.access.gpo.gov/su_docs/gils/gils.html>). Of the 8 participants who selected the social security administration database (S10a), all but 2 omitted the term "social security" from their search string; the participant who opted to search "all records" included the term and reported use of online help to acquire or refine searching skill (S1). It is interesting to note that only 2 users used the term "ordering," which is the operative "action" or "service" word in this concept. In addition, the three instances of "toll-free" may indicate an attempt at inappropriately high precision in specifying the type of information resource (i.e., only 1 user included the less-precise term "phone", which would not retrieve "telephone").
Recommendations: None. 

S19. [At GPO Access’ GILS and as a result of executing an unscripted query concerning a "toll-free number for ordering a statement of earned social security benefits"] How many "total hits" did GILS return?]
Findings: 9 participants answered this question. Results are shown below next to search terms used.

Discussion: By way of background and context, this nonscripted (i.e., user-directed) search exercise was included on the basis of an investigator’s recall of having used such a service more than 5 years ago. The investigators attempted several search strings within SSA’s GILS to locate the telephone number prior to the user session, and then replicated study participants’ queries using the above data—all to no avail. Acting on an impression that the benefits-statement service had been quite popular and may still be available, the investigator visited the USPS’s WINGS (Web Interactive Network of Government Services) <http://www.wings.usps.gov/Topten/> website, which is designed to provide direct access to frequently requested "pieces" of government information. Under WINGS’ "retirement" category one finds hypertext "Social Security Benefits/How do I…," which links to SSA’s "Personal Earnings and Benefit Estimate Statement" (PEBES) website <http://s00dace.ssa.gov/pro/pebes/pebes-home.shtml>. At this site, a user can request (via forms interface or email but apparently no longer by telephone) the subject benefits statement. When the "official" term Personal Earnings and Benefit Estimate Statement (no quote marks) was entered into SSA’s GILS, however, only one record was retrieved. This GILS record, entitled "Earnings Record And Self-Employment Income System," contains in the Purpose: "Master machine-readable file containing a summary of earnings for all individuals, including the self-employed, who pay social security taxes." However, this US Federal GILS "core" record also states that Availability is "none." (In addition, Record Source contains the word "none.") Unfortunately, a search of SSA’s GILS (which contains 1203 records according to the WAIS catalog) by use of "web* OR URL OR home* OR Internet OR http*" also failed to produce a record for the agency’s website, which may have led a user to the PEBES feature.
     Having confirmed the correct name for the PEBES service, the investigator analyzed the possibility that any of our study participants could have retrieved a GILS record if one existed by using their search strings. Only "social security benefit*" would have theoretically retrieved the record. Investigators’ attempt to replicate this user’s search (against "All records on GPO Access site," however, returned the default-maximum 40 hits, indicating that the user either failed to record his search strategy correctly or introduced a typographical error. The remainder of the search strings would have failed variously due to lack of truncation ("benefits" will not retrieve "benefit"), use of constraining quote marks, or use of terms based on an assumption of a telephonic rather than digital transmission medium.
Recommendation: See S22. 

S20. [At GPO Access’ GILS and as a result of executing an unscripted query concerning a "toll-free number for ordering a statement of earned social security benefits" Did you expect a "hit" that would obviously point you to the toll-free number?
Findings: 4 of 9 participants providing an answer to this question responded "yes"; the remainder did not expect a relevant hit.
Discussion: See S22.
Recommendations: See S22. 

S21. [At GPO Access’ GILS and as a result of executing an unscripted query concerning a "toll-free number for ordering a statement of earned social security benefits"] Do any "hits" appear to be relevant?
Findings: 4 of 9 participants providing an answer to this question responded "yes"; the remainder did not discern a relevant hit.
Discussion: Interestingly, only 2 of the 9 users had their "expectation of success or failure" fulfilled (i.e., in 7 cases S20 results mapped inversely to S21 results). See also S7 and S22.
Recommendations: See S22. 

S22. [At GPO Access’ GILS and as a result of executing an unscripted query concerning a "toll-free number for ordering a statement of earned social security benefits," if you did not expect a relevant hit or if you did not receive a relevant hit] Why not? (Circle all reasonable possibilities.)
Findings: 9 of 10 participants answered this question.

Discussion: Refer to S18 for an analysis of users’ search terms; "choice of database(s)" is moot given that all participants used either "All records on GPO Access site" or "Social Security Administration." Interestingly, of the 2 participants feeling "unsure of fielded searching," only one actually utilized that feature in the exercise. The options of (probably) "too new," "doesn’t exist," and "on a website" may be indicative of some uncertainty about policy and procedures for GILS content and its maintenance; S32j speaks to this issue as well. The investigators interpret the "long tedious list" comment to mean that the user did not wish to evaluate returned hits for relevancy (this user did not "expect" (S20) but did report (S21) at least one promising hit of the 40 produced). The user who "neglected to ask for ‘toll-free’ number" used "earned benefits" as a search string in the SSA GILS and received 3 hits (default "failed search" hits). The belief that the telephone number might be on a website (N=1) proved plausible (see S19).
Recommendations: Additional research into users’ attribution of error/failure cause will inform development and continuous improvement of online help facilities. 

S23. [After reading the GPO Access GILS-supplied definition of "control identifier"] Is the definition, and how it fits into GILS, clear to you?
Findings: 8 of 10 participants answered this question; 3 answered "yes" and 4 answered "no."
Discussion: The supplied definition was: "This element is defined by the information provider and is used to distinguish this locator record from all other GILS Core locator records. The control identifier should be distinguished with the record source agency acronym provided in the U.S. Government Manual." This definition is as published (in part) in the NARA Guidelines. While the present study did not ask users to try to pinpoint the source of confusion, we might assume the following. The word "element" means less to users than to GILS record creators, and users encounter this definition by pursuing links that refer to "fields" rather than elements. "Information provider" is also a vague term (e.g., a user could conceivably assume that it refers to database "owner," record creator, record source, the U.S. Public Printer, the agency’s public information officer, etc.). The definition might incorrectly assume an understanding or appreciation of the concept of "core" and "locator" records. Finally, users familiar with the contents of the U.S. Government Manual may be confused by the definition’s implication that the publication actually provides a "record source agency acronym." (S4 also addresses GILS nomenclature.)
Recommendations: See S4. 

S24. [Upon searching for a predetermined-as-duplicate control identifier at GPO Access’ GILS] Keeping in mind that you have searched the "control identifier" field, whose contents "distinguish this locator record from all other GILS Core locator records," what is you reaction to the list of hits?
Findings: 3 participants did not answer this question.

Discussion: Of the 5 users who responded negatively to S23 (definition was not clear to them), the 3 who also answered this question (S24) affirmed this confusion. The participants who attributed duplication to "core locator record" status and record-creator error both answered S23 affirmatively (definition was clear). The participant who selected the multilingual explanation responded to S23 negatively (definition was not clear). It is interesting to note that no users assumed system error.
Recommendations: If a purpose of control identifiers is to uniquely identify all records contained in all GILS (with the result of absolutely precise retrieval in known-item searching), a mechanism such as input masks or prevent-duplicates indexing feature should be implemented by all GILS providers.

S25a. [As a result of linking via browser bookmark to "U.S. Environmental Protection Agency—Search the GILS Database] Are you surprised to find that EPA’s Government Information Locator Service looks different from GPO Access’ Government Information Locator Service?
Findings: 8 of 10 participants answered this question. 3 responded affirmatively and 5 negatively.
Discussion: One explanation for "surprise" might lie in the title of this page, which could be interpreted to mean something other than an agency-specific GILS (i.e., a user, especially if under the impression that there is only one integrated GILS database, could conceivably believe that this site is EPA’s rendition of GPO Access’ GILS resource). (See also S1, S18a, and S31). "Lack of surprise" might be attributable to websurfers expectation of variation (e.g., among agency homepages or among the .edu, .gov., and .org servers of the same government legislative databases.
Recommendation: All nonbrokered agencies’ sites should be entitled "[Agency/Bureau/Etc. Name] GILS" rather than "GILS" or in some manner make apparent that they are a subset of the GILS universe. 

S25b. [As a result of linking via browser bookmark to "U.S. Environmental Protection Agency—Search the GILS Database] Do you expect the EPA’s Government Information Locator Service to operate exactly like GPO Access’ Government Information Locator Service?
Findings: Of the 8 participants responding, they were evenly divided between "yes" and "no."
Discussion: This finding perhaps corroborates the interpretation for S25a in that users appeared to manifesting some confusion about standard operability and/or agency leeway. (EPA’s GILS, in fact, does not "operate exactly like GPO Access’ GILS"—for example it, does not return search terms with results, it contains a "browse" feature, and offers a different set of elements for fielded searching.)
Recommendation: More extensive cross-GILS research could reveal whether diversity in presentation/operability approaches is a strength or weakness for end-users. 

S25c. [As a result of linking via browser bookmark to "U.S. Environmental Protection Agency—Search the GILS Database] Do you think that EPA’s Government Information Locator Service is part of GPO Access’ Government Information Locator Service?
Findings: Of the 8 users answering this question, 5 answered affirmatively and 3 negatively.
Discussion: The term "part of" might have been construed as "included on" the GPO GILS site (which it is not), or as "in cooperation with" GPO. Users also might have been under the impression that GPO is "in charge" of the GILS initiative and interpreted "part of" to mean "under the aegis of."
Recommendations: See S1. Also, popular government-information-seeking starting points (as determined by log transaction analysis, but we may assume the White House homepage, Library of Congress homepage, and GPO Access for examples) should consider featuring a link to an information-space map of GILS that shows host overlaps and organizational relationships. 

S26. [As a result of linking via browser bookmark to "U.S. Environmental Protection Agency—Search the GILS Database] What might you expect EPA’s Government Information Locator Service to provide? (Circle all that apply).
Findings: (8 of 10 participants responded to this question.)

Discussion: At this point in the script, participants had executed five searches at GPO Access’ GILS: three against "All records" and two against specific agencies’ databases. Two of the five searches were fielded and the remainder "full-text." These searches resulted in users’ examination of a minimum of three GILS records.
     The overall result of this question indicates some degree of understanding GILS to be a "locator of locators" (all participants expected a publications catalog). However, about two-thirds of the users also expressed expectation of actual documents (the "information itself") such as regulations and testimony even though they had no precedence for this belief in practice. Interestingly, only 1 of the 5 users expecting "full-text of EPA regulations" had expected an "online information locator" to contain "full-text of documents containing the information I seek" in P8. In addition, the notion that EPA GILS might provide census data or an inaugural address indicate confusion about the agency-specific content of each GILS database and may be related to information-space disorientation: 3 of the 5 users who thought EPA GILS was "part of" GPO Access’ GILS (S26c) expected to find census data on EPA GILS.
     In summary, with this limited line of inquiry, we may conclude that users reluctantly abandon an expectation of direct access to actual documents/resources and/or do not adopt readily to the GILS system of metadata records.
Recommendation: Caveats about content (such as that provided on FedWorld’s GILS site <http://www.fedworld.gov/gils>: "Please also note that GILS records are intended to allow you to learn about what government information is available, not to be the information that you might be seeking!" appear to be warranted. 

S27. Please rate how confident you would feel using the following [EPA GILS fielded search] options presented on this screen:
Findings: 8 of 10 participants answered this question.

Discussion: Of the options presented, Local Subject Term and Agency Program appear in item S3 (similar assessment of terms at GPO Access’ GILS), but users’ assessment of confidence declined during the intervening GILS experience. It is noted as well that, while S3 did not assess this option, GPO Access’ GILS equivalent phraseology for EPA’s GILS "complete text" is full text. The word choices for both systems appears to be risky in light of users’ expectation to access actual "information itself" (S8 and S26).
Recommendations: In addition to those provided in S8 and S26, it is suggested that the terms such as "complete text" and "full text" as search options be replaced with "all fields." 

S28. [As a result of scripted (directed) retrieval and "scanning" of an EPA GILS record entitled "Index to the Wetland Educational Resources distributed by the New England regional office, EPA"] Which of the following does this record describe?
Findings: 7 of 10 participants answered this question; 1 participant indicated 2 responses.

Discussion: For context, the subject record is reproduced in Attachment E3-1. The GILS record content analysis (see Appendix E-2 Record Content Analysis Findings, Discussion, and Recommendations) addresses resource description and information object identification in detail. However, the results of the present assessment indicate that record titles are powerful and that aggregation of resources within a single GILS record may be problematic. It is believed that the subject record describes "miscellaneous training items available separately." The concept of a "kit" however, may be inferred from the record’s statement that "Supplemental Information: Information collection [investigators’ emphasis] has a particular emphasis on wetlands stewardship materials for educators teaching grades K-12." Further, the "other" comments by these users corroborate qualitative results from the record content analysis indicating that "agency program," "purpose," "information resource" elements’ content are not consistently distinguishable in GILS records (see Appendix E-2 Record Content Analysis Findings, Discussion, and Recommendations). It is possible that the participant who suggested "like an infoguide" inadvertently described a result of aggregation rather than the actual materials.
Recommendations: The Resource Description element should be mandatory and its content be drawn from a controlled thesaurus. (See Appendix E-2 Record Content Analysis Findings, Discussion, and Recommendations for one approach to operationalizing information object/container terminology.)  

S29. How up-to-date are the described materials [described in the EPA record] ?
Findings: The following responses were received from this fill-in-the-blank question:

• December 6, 1995
• 1995 December 6
• 12/6/96 — over 1 yr old
• from 1995 I think
• don't know
• hard to tell from site
• not supplied

Discussion: About half of the participants answering this question referred to the record’s Date of Last Modification element (see Attachment E3-1)—a mistake common with GILS records creator according to the GILS record content analysis (see Appendix E-2 Record Content Analysis Findings, Discussion, and Recommendations). One of these participants offered what might be a value judgment in addition to the requested answer: "over one year old." In fact the record states "Time Period of Content: not supplied," which does refer to the information resource itself. Investigators assume that the participant who responded "hard to tell from site" (interpreted as "record") might have expected a "date of publication" field (which investigators’ acknowledge as not equivalent).
Recommendations: Adopt more precise nomenclature for Data of Last Modification (e.g., "GILS Record Revision Date" and "Time Period of Content" (e.g., "Time Span of Featured Subject Matter"). In addition, it is recommended that "publication dates" be required as available, and that historical resources cross-reference current ones as available. 

S30a. [After scripted retrieval and examination of a record from EPA GILS search page] Would you assume true/false: Information in EPA GILS is authored by EPA
Findings: 7 of 10 participants answered this question. 5 answered "true," the remainder "false."
Discussion: Some users are uncertain of "database owner," "URL host," "GILS Provider," and other responsibility and authority boundaries.
Recommendation: See S1.

S30b. [After scripted retrieval and examination of a record from EPA GILS search page] Would you assume true/false: Information in EPA GILS is the most current available
Findings: 7 of 10 participants answered this question. 4 answered "true," the remainder "false."
Discussion: Users do not universally assume that information on the Internet is current.
Recommendation: See S29. 

S30c. [After scripted retrieval and examination of a record from EPA GILS search page] Would you assume true/false: EPA is mandated by law to provide the information in GILS
Findings: 7 of 10 participants answered this question. 5 responded "false," the remainder "true."
Discussion: Study participants were not exposed to any form of GILS policy documentation during the session, and the 1 participant having experienced a previous GILS encounter (P3b) did not provide a response to this question. On this basis, investigators conclude that these responses represent guesses, and no conclusions may be drawn.
Recommendations: It would be interesting to pursue whether users informed of GILS purposes and mandate assess GILS differently—i.e., whether inconsistencies and errors are less tolerated.

S30d. [After scripted retrieval and examination of a record from EPA GILS search page] Would you assume true/false: EPA GILS describes every EPA publication
Findings: 7 of 10 participants answered this question. Only 1 participant answered "true," the others "false."
Discussion: Given that the scripted activities provided no direct indication of the number of records in this agency’s database, the finding indicates that users may appreciate the unfeasibility of describing all publications, much less "information resources" in a GILS database.
Recommendations: The concept of "Core" records should be re-specified or abandoned, as it is the sole yardstick for grasping the extent of Federal information resource holdings described in GILS. In addition, agencies should state on their GILS site what criteria how resources are chosen for description by a GILS record (S30j also speaks to this issue).

S30e. [After scripted retrieval and examination of a record from EPA GILS search page] Would you assume true/false: A duplicate of EPA GILS exists on GPO Access’ GILS
Findings: 7 of 10 participants answered this question. 5 responded "true," and the remainder "false."
Discussion: The participants had examined GPO Access’ GILS scrollbox of participating agencies to answer S1 and used the scrollbox in executing various searches. The EPA GILS page they encountered per the script did not refer to GPO Access’ GILS. In light of these observations, we could conclude that users may assume (perhaps by virtue of recalling an option for "All records" in the GPO Access GILS scrollbox) that all agencies’ GILS are served by GPO. (S31 addresses this perception more directly).
Recommendation: See S1.

S30f. [After scripted retrieval and examination of a record from EPA GILS search page] Would you assume true/false: I can find EPA GILS records by use of a web search engine such as Yahoo!, Alta Vista, or Lycos
Findings: 7 of 10 participants answered this question. 5 responded "true," the remainder "false."
Discussion: Users may assume that Web search engines automatically index all Internet content or that GILS database providers/owners have "registered" their content with popular search engines.
Recommendation: In GILS marketing, make this a "plus" (i.e., state that only through use of GILS can users directly access descriptions of thousands of agency resources). In addition, GILS homepages should be registered with popular Web search engines.

S30g. [After scripted retrieval and examination of a record from EPA GILS search page] Would you assume that this GILS has only information resources of EPA Headquarters in Washington, and does not include regional offices?
Findings: 7 of 10 participants answered this question. 5 participants believed this statement to be "false," 2 believed it to be "true."
Discussion: The question was included as a measure of participants’ recall that the title and other elements of the retrieved record stated clearly that the EPA information resource was "regional." Some users may require a more direct disclaimer as to the organizational scope of GILS records.
Recommendations: GILS sites should state the scope of the records collection in terms of, among other characteristics, organizational boundaries. 

31. [After approximately 1 hours’ GILS experience comprising 5 searches of GPO Access’ GILS (both "All records" and selected agency database(s) options) and 1 search on EPA GILS] How many GILS do you think may exist on the Internet today? (Circle one.)
Findings: 9 of 10 participants answered this question.

Discussion: After approximately 1 hours’ GILS experience comprising 5 searches of GPO Access’ GILS (both "All records" and selected agency database(s) options) and 1 search on an independent (nonbrokered) GILS, the finding that two-thirds of participants selected option other than "one for each Federal agency" indicates uncertainty about GILS scope, placement in information space, and/or "ownership."
Recommendations: See S1. The script assumed that information seekers are generally not motivated to link to and digest documentation concerning system or service policy. It was noted during development of the user session and record content analysis that a high number of agencies have mounted OMB 95-01 and other policy documentation, presumably by way of informing visitors about GILS rationale. This practice should be supplemented by an educating document that contains a standard (i.e., government-wide) statement about the GILS universe and the host’s placement within it. 

S32a. [After 1 hour’s experience comprising 6 scripted searches on GPO Access’ and EPA’s GILS plus up to 10 minutes nonscripted exploration of up to 5 nonbrokered GILS]
All agencies’ GILS should be searchable together, from one website.
Findings: 9 of 10 participants answered this question.

Discussion: Centralization of access is a desired state.
Recommendations: None. 

S32b. [After 1 hour’s experience comprising 6 scripted searches on GPO Access’ and EPA’s GILS plus up to 10 minutes nonscripted exploration of up to 5 nonbrokered GILS]
There are not enough fields to search in a GILS database.
Findings: 9 of 10 participants answered this question.

Discussion: Of participants expressing an opinion, there are a sufficient number of fields to search.
Recommendations: Assess fielded-search usage in actual practice by means of log transaction analysis.

S32c. [After 1 hour’s experience comprising 6 scripted searches on GPO Access’ and EPA’s GILS plus up to 10 minutes nonscripted exploration of up to 5 nonbrokered GILS]
All government documents on the Internet should be hotlinked from one electronic card catalog.
Findings: 9 of 10 participants answered this question.

Discussion: This question assumes "full document retrieval" or "access to the actual information." Respondents by a reasonable margin prefer "one-stop shopping."
Recommendations: Implementation of GILS on Z39.50-compliant servers and increased description of online resources will promote a perception of "seamless" service for GILS.

S32d. [After 1 hour’s experience comprising 6 scripted searches on GPO Access’ and EPA’s GILS plus up to 10 minutes nonscripted exploration of up to 5 nonbrokered GILS]
It would be easier to search GILS records if they were grouped hierarchically by subject.
Findings: 9 of 10 participants answered this question.

Discussion: This result of two-thirds "agreeing" to the serviceability of subject-oriented access is interesting. GILS is based on the GPO model of agency name as primary access, as is most of the White House website. More than half of our well-educated demographic (P5) indicating a lack of awareness of agency functions (S32e) points to the need for alternative approaches to locating government information.
Recommendations: Further research appears warranted in two areas: the incidence of subject-oriented inquiry vis-a-vis GILS and the feasibility of a non-organization-based classification scheme. Also see S19.

S32e. [After 1 hour’s experience comprising 6 scripted searches on GPO Access’ and EPA’s GILS plus up to 10 minutes nonscripted exploration of up to 5 nonbrokered GILS]
I am unaware of the function or purpose of many Federal agencies.
Findings: 9 of 10 participants answered this question.

Discussion: From this nearly evenly divided result from a well-educated demographic (P5), we may conclude that GILS implicit assumption of agency-mission knowledge is unwarranted. See S32d.
Recommendations: Agency GILS "index.htm" or search pages could provide a prominent link to their mission statement and/or a list of general functions (perhaps from the U.S. Government Manual).

S32f. [After 1 hour’s experience comprising 6 scripted searches on GPO Access’ and EPA’s GILS plus up to 10 minutes nonscripted exploration of up to 5 nonbrokered GILS]
All GILS records should look alike.
Findings: 9 of 10 participants answered this question.

Discussion: Participants expressed a "soft" preference for consistency in the appearance of GILS records.
Recommendations: Further research could operatonalize "look alike"—e.g., determine whether this preference considers file format (HTML, PDF, ASCII, etc.), presentation attributes (indentation, boldface type, etc.), and/or a uniform "template" of all elements (populated or not). The GILS record content analysis addresses this issue in more detail (see Appendix E-2 Record Content Analysis Findings, Discussion, and Recommendations). 

S32g. [After 1 hour’s experience comprising 6 scripted searches on GPO Access’ and EPA’s GILS plus up to 10 minutes nonscripted exploration of up to 5 nonbrokered GILS]
I will use GILS to locate government information in the future.
Findings: 9 of 10 participants answered this question.

Discussion: On the average, participants anticipate using GILS again. The "strongly agree" response was from the same participant "strongly agree"ing that "GILS is an efficient service" in S32h; in fact, all but 1 participant’s responses mapped positively from future-use to efficiency of service.
Recommendations: User assessments should be followed up with a questionnaire concerning subsequent GILS vs. other government NIDR tool usage, discussion (positive or negative) of GILS with others, etc. within 6 to 8 weeks of the original session. 

S32h. [After 1 hour’s experience comprising 6 scripted searches on GPO Access’ and EPA’s GILS plus up to 10 minutes nonscripted exploration of up to 5 nonbrokered GILS]
GILS is an efficient service.
Findings: 9 of 10 participants answered this question.

Discussion: By a narrow call, participants considered GILS "efficient." It is noteworthy, however, that no one "strongly disagreed" to this question. The results of this question do not appear to correlate with those of S32l (GILS is an improvement over microfiche and paper indexes), indicating that participants judged "efficiency" relative to other networked information resources.
Recommendations: Further research could operationalize "efficient" for various user communities and types of information need; see S19.

S32i. [After 1 hour’s experience comprising 6 scripted searches on GPO Access’ and EPA’s GILS plus up to 10 minutes nonscripted exploration of up to 5 nonbrokered GILS]
GILS probably helps agencies manage their information resources.
Findings: 9 of 10 participants answered this question.

Discussion: Bearing in mind that users did not encounter GILS policy documentation, it may be assumed that exposure to certain element names such as "control identifier" and "schedule number" led some participants to perceive an IRM objective within GILS.
Recommendations: None.

S32j. [After 1 hour’s experience comprising 6 scripted searches on GPO Access’ and EPA’s GILS plus up to 10 minutes nonscripted exploration of up to 5 nonbrokered GILS]
It is clear to me how agencies choose what to include in GILS.
Findings: 9 of 10 participants answered this question.

Discussion: Participants were not clear as to the criteria for inclusion of an information resource in GILS collections. Of the Likert scale questions at the end of the script, this question evoked the strongest negative response. See also S26 and S30b,c,d,e.
Recommendations: See S26.

S32k. [After 1 hour’s experience comprising 6 scripted searches on GPO Access’ and EPA’s GILS plus up to 10 minutes nonscripted exploration of up to 5 nonbrokered GILS]
Public electronic access to government information is important.
Findings: 9 of 10 participants answered this question.

Discussion: Our well-educated study demographic (P5), expressing a variety of government information needs (P6) clearly places importance on public access.
Recommendations: See S22; key to "access" is knowledge of availability. GILS marketing efforts should promote "the right to know."

S32l. [After 1 hour’s experience comprising 6 scripted searches on GPO Access’ and EPA’s GILS plus up to 10 minutes nonscripted exploration of up to 5 nonbrokered GILS]
GILS is an improvement over microfiche and paper indexes.
Findings: 9 of 10 participants answered this question.

Discussion: Study participants consider GILS preferable to microfiche and paper indexes. The 2 users expressing "no opinion" searched both paper and online sources of government information relatively infrequently (P2a and P2b).
Recommendations: Further research could determine what features make GILS a more appealing locator tool—i.e., the ability to search across information providers (GPO Access GILS "All records" option), the ease of accessing GILS via Internet vs. a visit to a physical library, the degree of indexing, etc. These features should then be predominant in GILS marketing.

S32m. [After 1 hour’s experience comprising 6 scripted searches on GPO Access’ and EPA’s GILS plus up to 10 minutes nonscripted exploration of up to 5 nonbrokered GILS]
All GILS records should contain information in all fields.
Findings: 9 of 10 participants answered this question.

Discussion: This result indicates a fairly strong consensus that all GILS elements should be populated, but no inferences can be drawn from the current study as to why. No correlation is apparent between participants’ support of field-population and their self-confidence in fielded searching (S22).
Recommendations: It is recommended that GILS designers perform a confirmatory analysis that fielded searching improves retrieval precision in GILS for both known-item and exploratory searching.

S32n. [After 1 hour’s experience comprising 6 scripted searches on GPO Access’ and EPA’s GILS plus up to 10 minutes nonscripted exploration of up to 5 nonbrokered GILS]
The quality of the records I examined varied widely.
Findings: 9 of 10 participants answered this question.

Discussion: One-third of respondents to this questions reported a wide variation in record quality. The GILS record content analysis addresses issues of quality in depth (see Appendix E-2 Record Content Analysis Findings, Discussion, and Recommendations).
Recommendations: Additional user-centered research is encouraged in order to operationalize "quality" criteria and develop and/or choose existing records as model(s) for benchmarking. 

5.0. SUMMARY OF RECOMMENDATIONS 

The following recommendations have been extracted from Section 4.0 Findings, Discussion, and Recommendations, and are arranged according to opportunities for increasing GILS acceptance among user communities. Many of the proposals can be implemented at the agency level, while others demand inter-agency consensus and cooperation. It is strongly recommended that regardless of the level of effort, agencies work toward adopting standard practices for GILS service features and record characteristics to enhance users’ orientation in information space and promote GILS as a government-wide program.

5.1. Increase Users’ Searching Confidence 

• Present concise, comprehensible search instructions on the same page as the search input mechanism. Provide an example.
• Subject the element definitions presented to users to reading level (e.g., Flesch Reading Ease, Flesch-Kincaid Grade Level, etc.) and grammar checks to compute "fog index" by identification of incomplete clauses, jargon, passive voice, characters/syllables per word and words per sentence.
• Implement a record input mask or prevent-duplicates indexing feature to avoid application of the same Title or Control Identifier to more than one record.
• Avoid use of "complete text" and "full text" terminology in search options; use "all fields" instead.
• Provide a prominent link from agency GILS "index.htm" or search page to their mission statement and/or a list of general functions (perhaps from the U.S. Government Manual) to reduce dependency on an assumption of user cognizance.
• State clearly and prominently on each GILS site (ref: FedWorld) "Please also note that GILS records are intended to allow you to learn about what government information is available, not to be the information that you might be seeking!"
• Avoid linking to, or quoting verbatim, Federal information policy documentation for the purpose of introducing users to GILS functionality. 

5.2. Improve GILS Niche In Information Space 

• Entitle all record-source (nonbrokered) agencies’ sites "[Agency/Bureau/Etc. Name] GILS" rather than "GILS" or in some manner make apparent that they are a component of the GILS universe.
• Include a straightforward statement on the GPO Access GILS search page(s) to the effect that the databases contain an agency’s GILS records of information resources created and available from that agency and that GPO GILS listing may be incomplete.
• Determine the most effective way to convey GILS in the singular (as an agency database) vs. GILS as the collective.
• Provide a GILS hyperlink from popular government-information-seeking starting points (as determined by log transaction analysis. but we may assume the White House homepage, Library of Congress homepage, and GPO Access for examples).
• Create an information-space map of GILS that shows host overlaps and organizational relationships.
• State the scope of each GILS record collection in terms of, among other characteristics, organizational boundaries on each GILS site.
• Implement GILS on Z39.50-compliant servers and increase description of online resources to promote a perception of "seamless" service.
• Advertise in government publications and libraries; incorporate GILS linkage into agency and White House homepages; register/index GILS homepages with popular web search engines.
• Capitalize on GILS "exclusivity" from regular webpages (i.e., state that only through use of GILS can users directly access descriptions of thousands of agency resources). 

5.3. Improve GILS Efficacy in NIDR and Revise NARA Guidelines Accordingly 

• Develop and require a Resource Description element whose value is recognizable to the user, rather than the distributor, and is drawn from a controlled thesaurus; Appendix C-4 Record Content Analysis Methodology provides for one approach to operationalizing information object/container terminology .
• Adopt more precise nomenclature for Date of Last Modification (e.g., "GILS Record Revision Date" and "Time Period of Content" (e.g., "Time Span of Featured Subject Matter" vs. "Publication Date").
• Re-specify or abandon the concept of "Core" records, as it is the sole yardstick for grasping the extent of Federal information resource holdings described in GILS. 

5.4. Improve the Quality and Consistency of GILS Records 

• Enlist objective content reviewers to evaluate conformance of fields’ content to qualitative descriptions and examples provided in the NARA Guidelines.
• Select or develop "model" records, as determined through user-satisfaction studies, to serve as an example to record creators and as a benchmark for evaluators.
• Enlist an objective party to evaluate hyperlinks to ensure that the record creator is not over-assuming an intellectual "hop" vis-a-vis the content of the linked-to page.
• View product prior to mounting as displayed by browser(s) selected on the basis of published usage reports (e.g., "Browser Battle" July 1996 Internet World p. 40) or agency website access log analysis. 

6.0. AREAS FOR FURTHER RESEARCH 

Results of the online scripted user assessment presented the following areas of exploration to increase GILS responsiveness and operational serviceability for users. 

• Determine user satisfaction with agency name as the primary access point or as a starting point in exploring subject-oriented access to GILS systems
• Elucidate relationships between longevity of Internet experience, self-ratings of "familiarity" or "proficiency," and satisfaction with system results to inform the level of online help or hands-on training offered.
• Investigate the nature of users’ information needs for government information by questions such "what was on your mind the last time you recall deliberately searching for or monitoring government information." For example, users seeking to satisfy a specific and direct but occasional information need may prefer the approach planned by USPS’s WINGS (Web Interactive Network of Government Services) <http://www.wings.usps.gov/Topten/>, which will present information on, among other things, tax-return filing, requesting birth certificates, and employment opportunities.
• Perform server transaction log analysis to inform decisions about presentation of metadata elements for fielded searching and results.
• Survey the agencies that are using controlled vocabulary and determine, through log transaction analysis, whether the practice increases retrievals.
• Capture GILS user reactions, attributions of error/failure cause, and use of metadata elements through "talk-aloud" protocol.
• Analyze the extent to which fielded searching in GILS improves for both known-item and exploratory searching.
• Determine whether diversity in record display characteristics (file format, layout, etc.), results presentation, and searching features among agencies is a strength or weakness for end-users.
• Follow up all user sessions with a questionnaire concerning subsequent GILS vs. other government NIDR tool usage, discussion (positive or negative) of GILS with others, etc. within 6 to 8 weeks of the original session.
• Determine what features make GILS a more appealing locator tool (i.e., the ability to search across information providers [GPO Access GILS "All records" option]), the ease of accessing GILS via Internet vs. a visit to a physical library, the degree of indexing, etc.—and capitalize on these features in marketing. 

7.0. CONCLUSION 

The online scripted user assessments produced data that confirmed several important findings from other public user-oriented data collection activities such as focus groups. 

GILS is not perceived as easy to use, predictable, or efficient in terms of satisfying information needs. Users perceive the use of bibliographic terminology and the lack of straightforward search instructions as uninviting, the content of the databases as unknowable, the service and record quality as uneven, and the lack of full text as approaching unforgivable. 

Nonetheless, the results of the sessions and subsequent debriefings show that users believe in GILS potential if not all its current implementation characteristics. Further deployment of scripts to assess factors evoking both delight and disappointment among a variety of user communities—including librarians, publishers, public-information activists, business owners, and researchers—is recommended as an effective mechanism for gaining feedback from the "front line" in order to achieve that potential.

Title:
Index to the Wetland Educational Resources distributed by the New England regional office, EPA 

Acronym:
Not supplied 

Originator:
Environmental Protection Agency
Region 1
Wetland Protection Section 

Controlled Vocabulary (Library of Congress Subject Headings):
Aquatic ecology; Conservation of natural resources; Biological diversity conservation; Biotic communities;
Document delivery; Ecology; Environmental education; Environmental protection; Government publications;
Wetlands; Wildlife 

Controlled Vocabulary (Terms of Environment):
Habitat 

Controlled Vocabulary (Supplied by GILS cataloger):
Educational materials 

Local Subject Term:
US Federal GILS; wetlands; habitat protection; biodiversity

Abstract:
The U.S. Environmental Protection Agency's New England regional office produces and distributes numerous
wetlands educational materials to the public — booklets, fact sheets, videocassettes, posters, etc. These materials
cover a number of topics — what wetlands are, why they are important, how citizens (particularly students) can
protect wetlands and the federal regulations that protect them. Materials are produced with specific target audiences
in mind, including students, teachers, municipal officials, and developers.  

Purpose:
The wetlands educational materials have been developed to increase the public's awareness of the importance of
wetlands and how federal wetland regulations protect wetlands.  

Agency Program:
Not supplied  

Spatial Reference:
Geographic Keyword Name (Library of Congress Subject Headings):
New England  

Spatial Reference:
Geographic Name (Hazardous Waste Superfund Database Thesaurus):
Region 1  

Time Period of Content:
Time Period-Structured:
Not supplied
Time Period-Textual:
Not supplied  

Availability:
Distributor:
     Name: Wetland Protection Section
     Organization: U.S. Environmental Protection Agency
     New England Regional Office
     Street Address: JFK Federal Building
     City: Boston  
     State: MA
     Zip Code: 02203
     Country: USA
     Hours of Service: 8:30 a.m. - 5:00 p.m. (EST) M - F
     Telephone: 617-565-4421
     Fax: 617-565-4940  

Resource Description:
Not supplied  

Order Process:
Materials may be ordered for free by writing to the address listed above.  

Technical Prerequisites:
None  

Available Linkage:
Not supplied  

Available Linkage Type:
Not supplied  

Sources of Data:
U.S. Environmental Protection Agency, environmental education nonprofit organizations, U.S. Army Corps of
Engineers, other
federal agencies  

Access Constraints:
None  

Use Constraints:
None  

Point of Contact:
     Name: Wetland Protection Section
     Organization: U.S. Environmental Protection Agency
     New England regional office
     Street Address: JFK Federal Building
     City: Boston
     State: MA
     Zip Code: 02203
     Country: USA
     Network Address: Not supplied
     Hours of Service: 8:30 a.m. - 5:00 p.m. (EST) M - F
     Telephone: 617-565-4421
     Fax: 617-565-4940  

Supplemental Information:
Information collection has a particular emphasis on wetlands stewardship materials for educators teaching grades K-
12. All materials are located at the EPA New England regional office building, 1 Congress St., Boston, MA. For
information on wetlands materials available throughout the country, contact the U.S. Environmental Protection
Agency wetlands hotline at (800) 832-7828.  

Schedule Number:
Not applicable  

Control Identifier:
EPA/GENERAL01003  

Record Source:
Environmental Protection Agency
Region 1
Wetland Protection Section  

Date of Last Modification:
19951206  

URL: http://www.epa.gov/earth100/records/g01003.html

Next Appendix
Previous Appendix
Table of Contents