November 15, 1997
William E. Moen
CIMI Z39.50
Interoperability Testbed Meeting
October 13 & 15, 1997
St. Louis, Missouri
MEETING SUMMARY
Attending
Testbed Participants
Project Staff
William E. Moen, Project Manager, Meeting Facilitator
John Perkins, CIMI Executive Director
Kody Janney, Consultant
Observers
Overview
This two-day meeting marked the final phase of the six-month CIMI Interoperability Testbed. The purpose of this meeting was to summarize the status of testbed implementations, identify problems, issues, and deficiencies in the Implementors Agreement (August 29 version), and suggestion revision paths for the CIMI Profile. In particular, the meeting discussed attribute set, access points, schema, and other related issues. Finally, the meeting had as a goal gaining an indication of implementors interest in ongoing testbed work.
The agenda for the meeting is available at: http://www.cimi.org/documents/z_final_agenda.html
The following provides a summary of discussions and activities during the meetings.
Day One: October 13, 1997 (9pm–5pm)
Much of the first day was spent with testbed participants reporting on the status and results of their implementations. In part, this reports provided an indication of implementation and specifications problems that still needed attention. Moen noted a number if issues with the Implementors Agreement (August 29, 1997 version; all references to the Implementors Agreement in this summary point to the Aug. 29 version). Also, the implementors provided some indication as to their experience with the entire testbed experience and suggestions for future activities.
FINAL IMPLEMENTOR STATUS REPORTS AND DEMONSTRATIONS
Blue Angel Technologies
Jeff Tanara reported that the work on the client was completed. He had tested the client successfully with the following testbed servers: Crossnet, Databasix, ELISE/DeMontfort, ELISE/Tilburg, Finsiel, Forth, Joanneum. The client is configured to support all the attribute types and values specified in the Implementors Agreement.
In terms of the actual testbed experience, Tanara suggested that he would have been helped in his development through more feedback in the early part of the project. He was aware that many server implementations did not get completed until relatively late in the testbed period. This meant that there a compressed time period when most of the activity took place. For dealing with problems with his client and others' servers, he worked most of these out via email. He did comment on the lack of information on the various servers. He needed more complete information about the servers in a more timely, systematic manner.
He then demonstrated the client and its interaction with several of the servers.
A number if implementors suggested/identified needed features or problems:
Databasix
Chris Turner and Ronald Van Der Meer reported on Databasix implementation. It is complete and supports both SUTRS and USMARC record syntaxes.
In terms of problems and issues from the testbed, they reported that working with the data was somewhat difficult because the database structure they had was very flat, and mapping into the GRS structure was a challenge. They spent considerable time getting the data in a shape that could be used.
Walter Koch noted that in his implementation, his data were much more highly structured than that in the GRS record. Michael Selway noted that putting data into the structure had not been too difficult.
ELISE/DeMontfort University & SSL
Michael Selway (SSL) and John Eyre (ELISE/DMU) reported on their efforts. Selway noted that the CIMI server toolkit is being used by three of the testbed participants: DMU, NTU (Taiwan), and FORTH (Greece). In addition, several of the other participants are using the server toolkit for reference, looking at the code, etc.
SSL is preparing a new release of the server toolkit, and it should be released as soon as the documentation on it is done. The new release will address the structured record as required by the Implementors Agreement, including the repeatability of some of the elements. In addition, the release will allow the client to pass the request for a record syntax to the server.
ELISE/DMU hosts the SSL toolkit and is providing the ELISE data.
Selway suggested that he will be continuing work on the Aquarelle project and noted that there will be a need to fold the Aquarelle Profile and the CIMI Profile back together. Based on the requirements of the Aquarelle Project, its Profile has specifications not in the CIMI Profile. And the CIMI Profile has evolved since the inception of the Aquarelle Project.
Joanneum
Walter Koch reported on the status of his server and client implementations. His server has a relational database system behind it. He is offering two sets of data. The client was first based on the Stanford client, but now uses YAZ. The client can accept USMARC records.
Koch identified a number of key issues and topics to consider:
Crossnet
Martin Bush reported on Crossnet's implementation. The data are from a fairly small museum. In an Access database (approximately 500 records). Crossnet had to scan photos supplied by the museum. He noted that one problem they faced was with incomplete data.
In addition the server, Crossnet also implemented a client (although not completely CIMI compliant); the client was developed as part of the ICONE. It supports SUTRS and USMARC, but not GRS.
One issue Bush identified was the length of the brief record -- that there's too much in there.
Finsiel
Luca Lelli provided the status report on Finsiel accomplishments. Currently the server provides access to a subset of records from the Consortium Alinari collection of photographs. He hopes to add more records from the Alinari collection and from another database soon.
Identifying some issues and problems, Lelli suggested that the brief record was too long, too many fields. In addition, he noted that the Structure Attribute is mandatory, but does not think this is necessary. This can be changed in the revised profile.
As a participant in the Aquarelle Project, he said there was some problem in coordinating the work and efforts for both since there are differences in the two profiles.
CHIN
Rob Dallas reported on CHIN's server. The server is up and running, but it does not support GRS. It supports only SUTRS. He has been working on GRS support and has put in several weeks of effort so far.
As one of the problems, he noted that mapping the attributes was not straightforward.
Also, he had experienced some problems with the TCP port number, and because of firewall protections, he did not want to use the regular Z39.50 port (210?). He suggested that all testbed participants use a port number between 2100 and 2110. All the participants agreed to do this as of November 1.
Willoughby
Larry Mills-Gahl reported on Willoughby's efforts. No operating server at this point. Had started building a server using YAZ, but is now using the Blue Angel server toolkit. His current roadblock to progress is mapping the attributes and elements to the database. He is doing the mapping, but he noted that he is not the appropriate person to do this since he does not know the data as others in his organization do. He is waiting for approval on the mappings, etc. before he can make his server and data available.
Center for Excellence for Research in Computer Systems
Chung-Chen Lai and Chiou-Shann Fuh reported that NTU is continuing with its implementation activities. They realized that because of limited experience and knowledge about Z39.50, their progress had been slower than expected. They will continue their work, and will look forward to the new released of the SSL toolkit.
University of California
Mark Needleman reported that his client is running and his programmer has tested the client with many of the servers successfully.
Other Reports
Richard Schleicher from Questor commented on his interest in the work done to date in the testbed, and he stated that Questor is interested in becoming involved with the work. He noted that he had reorganized Questor and now has people in the right places to move ahead, specifically David Scarborough (director of new programs and R&D) and Bob Edberg.
ISSUES RELATED TO THE TESTBED AND IDENTIFICATION OF PROBLEMS/CHANGES TO IA, Part 1
Subsequent to the implementors status reports, the group discussed a series of issues and concerns related to the testbed experience and the Implementors Agreement. This segment was intended to help guide the revision of the CIMI Profile. The following summarizes the initial discussion on these identified issues. Several of these were returned to for additional discussion on Day 2. See under Day 2 for additional discussion on these and other issues.
NOTE: Moen developed a separate document at the end of Day 1 that summarized the various issues that came up in the meeting. This is available as Issues and Ideas From Day 1 Working Group Meeting
Problems Related to Data
At the time of the selection of participants for the testbed, each participant identified data (and sources of data, e.g., museum partners) they would bring to the testbed for search and retrieval. The extent of data actually offered, however, was not as large as originally intended. Through the status reports, several implementors discussed the amount of unanticipated work needed to build database and massage data to make it ready for Z39.50 access.
Some of the participants said the data they had was not structured enough in the database to easily map to the GRS structure. In those cases, they restructured the data into new databases to allow such mapping. While this may seen inimical to the spirit of Z39.50 (which should be able to address the data as it is), several participants saw the creation of new databases as not a major concern.
Another aspect of the data issue is in terms of mappings for both the attributes in the Attribute Set and the elements in the GRS abstract record schema. There was agreement that mapping guidance and crosswalks are essential for helping implementors do this. Because of the varieties of schemas (sometimes vendor-based and proprietary), simply mapping to a data model such as CIDOC may not be the answer. Thus, again, the need for crosswalks.
One suggestion for the crosswalks would be to start at the "top" with the most common, generic access points and database elements (or concepts related to those elements), and do the crosswalk for the attributes and elements for those. Continue "down" the list until one reaches the point where there is less common agreement among the various database schemas, access points, etc.
For dealing with the various database record fields that don't map into the relatively "common, generic" ones, one suggestion was to use "string tags" in the GRS record for those. There was also the suggestion to use SUTRS as an acceptable record syntax to address either highly structured or relatively flat data, simply to move the record from server to client.
[Moen note: I think the issue of whether/when data will need to be massaged or restructured is a research question that needs to be followed up. I don't think it should necessarily limit the CIMI Profile's approach to GRS and the abstract record structure. And I think with the crosswalk and mapping guidance, some of the other issues may become less of a barrier for implementors or potential implementors. Comments?]
Harmonization of CIMI Profile and Aquarelle Profile
Selway noted that the Aquarelle Profile was based on a very early version of the CIMI Profile. Since that time the CIMI Profile has evolved to address the needs and requirements of the testbed participants. The divergence of the two Profiles will need to be addressed, in particular the CIMI Abstract Record Structure. He said he would propose (as a participant in the Aquarelle work) that Aquarelle move towards the more highly structured CIMI retrieval structure.
In addition, he suggested that he would like to see the CIMI Profile address the issue of date ranges, and being able to search on date ranges.
He also noted that the Aquarelle defined several attributes, and he would like to see these pulled into the CIMI Attribute Set.
Size of Brief Record
Several participants commented that the "brief" record is anything but brief. One suggestion was for a brief record consisting of simply a couple of lines of information, and then let the use "drill down" into a more rich descriptive record. Another suggestion was to make the return and display of images in the brief record optional.
There was agreement that the brief needed to be briefer. The approach will be to define two element sets, where brief is large enough to address a client building a "tombstone" for the image. To figure out what the elements are that will be needed, there was the suggestion that we look at what museums are currently providing as "tombstone" information on their websites. At least one participant offered to provide Moen with what their system provides as a "tombstone."
Handling Images
One participant suggested that the Profile provide explicit guidance on handling renditions. For example, if the only rendition of an image is a relatively high resolution, it may not be appropriate to put that into a brief record. Currently, the renditions are returned in an ordered, with the first to be assumed the "smallest" rendition. If it is a thumbnail, that is not a problem. But if the first rendition is quite large, the client and user may not be prepared to deal with time involved in loading the image (especially if it is just one record in a large result set). The suggestion is to give explicit instructions that the client is to look at the appliedVariant information and then make some decision whether or not to display an image in the brief record.
The first day's meeting adjourned at the end of this discussion. John Perkins invited the participants to attend and participate in the CIMI Membership Meeting on Tuesday. Moen said that the testbed group would reconvene on Wednesday morning.
Day Two: October 15, 1997 (9pm–5pm)
Kody Janney, a consultant working for CIMI, led a lengthy discussion related to the crosswalk she is developing. The goal of the crosswalk is to map between the CIMI Attribute Set and CIMI Abstract Record Schema (ARS) to concepts of access points and database elements in other community "standards" such as REACH, AMICO, CDWA, etc. Part of the problem was to clear up semantics and redundancy in the CIMI Use attributes and ARS elements. During the interoperability testbed Use attributes and elements were added to a set of attributes and elements inherited from the original draft CIMI Profile (June 1996).
Janney went through her draft crosswalk, item by item to clarify the intent of the attribute or element, and to gain consensus on the utility or need for each. This section will not detail the discussion since this will be written up by Janney and Moen in an associated document. The decisions, however, for changes in the Attribute Set and Schema will be reflected in the revised CIMI Profile.
ISSUES RELATED TO THE TESTBED AND IDENTIFICATION OF PROBLEMS/CHANGES TO IA, Part 2
During the remaining discussion on Day 2, the group addressed a number of additional issues related to revising the profile. The following highlight key topics and issues.
Scope of the Revised Profile
Perkins suggested that the revised profile needs to be comprehensive enough to that when it is examined by the museum community they don't find obvious gaps in it. While the Profile is in the language of Z39.50, there needs to be enough material in the profile, its appendices, or additional documentation for them to recognize its utility and applicability.
Moen raised the question of the extent to which the revised Profile should include specifications that have not been tested in the interoperability testbed, e.g., all the SGML specs from the draft CIMI Profile (June 1996). Perkins recommended that the Profile needs to identify known issues, document decisions clearly, and also document where the Profile is not specifying Z39.50 use and why (e.g., with the SGML). Some of this could likely be in one or more appendices to the Profile.
The important goal is to specify Z39.50 in the Profile for what we think is appropriate and workable, and also to include ancillary documentation to explain in non-Z language, the purposes, functions, benefits, and limitations of the revised Profile.
Compliance with the Profile
Continuing the discussion related to Scope of Revised Profile, David Scarborough raised the issue of what it takes to be compliant with the Profile. For example, if the Profile requires the support of searching on a wide range of access points, but the data are not there, what is the server's level of compliance. Chris Turner and Scarborough suggested guidance in the Profile's section on compliance to suggest something along the lines of "you can use this Profile to the level at which you are willing to populate the database with appropriate data." [Moen's note: I have in my notes that David and Chris would send me language for this guidance. ]
Moen suggested that the Profile could lay out different levels of conformance and that these could be tied to the level of searching/retrieval. He suggested three levels:
Participants agreed that this was worth pursuing and Moen will propose in the revised Profile.
Who, Where, What, When Searching
Selway suggested that we could lay out a way to do a simple 4W search. Aquarelle is doing this for searching now. Perkins noted that the CHIO demonstrator provided this sort of search interface. Ways to approach this could be to define four new Use Attributes, but several participants suggested there might be retrieval problems with this. The Profile could provide guidance on mapping such searches to existing Use attributes.
Appendix C -- Combinations of Attributes and Values
Currently, Appendix C seems quite arbitrary and needs revision. This can be revised based on the discussion regarding levels of conformance/type of search and retrieval supported. We can make some statement about how in the testbed, a certain level of mandatory attributes and legal combinations were true, but also state that future and broader implementation experience can/should guide these specifications.
In part, it comes down to servers being able to support all the attributes for which they have data. Language like this could be put in the revised Profile.
Revising/Renumbering the Attribute Set and TagSet
Given the extent of revision of the CIMI Attribute Set (and possibly the TagSet/Abstract Record Structure) Moen asked if implementors would be severely impacted if he renumbered attributes and element tags. Generally, participants did not want this to be done. The agreement was to retain current numbers where possible. Where a use attribute is removed, keep the number there but designate it as [reserved], not [undefined].
Other Issues and Work Items
Selway suggested that future work items should include the following:
Another suggestion was for profiling the use of HTML and SGML in Z39.50.
FINAL SUMMARY OF WHERE TO GO WITH THE INTEROPERABILITY TESTBED
During the final time remaining for the meeting, the group discussed where to go next with the interoperability testbed, the CIMI Z39.50 working group, etc.
Moen will revise the Profile and have that ready for review later in the year.
Perkins suggested that there should be an ongoing Z39.50 Working Group within CIMI to address the work that remains to be done, and if the NEH provides a second round of funding to CIMI to extend its work in distributed search and retrieval and navigation of digital collections.
Turner said that such a group needs a program of work, and that if the group is only constituted for meeting to discuss issues, it may be too unfocused. He suggested a program of work approach with clear objectives and deliverables.
Selway said that there is a need for ongoing maintenance of the Profile. As it gets used more widely, there will be a need for a forum to deal with changes, etc.
Mill-Gahl suggested that there will be ongoing issues related to interoperability that such as group could address. Turner added that such a group should be "getting our hands dirty and doing it."
Dallas asked what Perkins expectations were of the currently deployed servers. Perkins said that NEH has requirements for seeing the implementations, the products of the work, and that it's clear to them that we did what we claim we did. That means keeping the servers available through the end of the project/completion of the final report and its delivery to NEH (end of first quarter 1998).
Turner pointed out that having the servers available is very important for encouraging new client development and providing a way of testing the clients.
Perkins summarized by saying that CIMI should maintain a Z39.50 Working Group that will undertake more profiling work, it will address more Dublin Core integration, and it can serve to evaluation the implementations, etc. He suggested that the implementors be polled for a list of work items they would be willing to commit time, resources, and energy to.
Next Meeting of the CIMI Z39.50 Working Group
Although Moen (with the support of several participants) tried to lobby that the next meeting be held in Texas, the group decided that the rational thing to do would be to schedule a meeting in conjunction with the January ZIG meeting in Orlando. Tentatively, the Working Group will plan to meet January 19-20 (Monday and Tuesday) prior to the ZIG meeting in Orlando. The meeting will do a final review of the revised CIMI Profile, any documentation developed for implementation guidance, the crosswalk and mappings. In addition, the group will discuss specific work items to undertake.
TO DOs:
1. Tanara will post info about the versions of the Java virtual machine, especially the new version of JDK, which will allow the client to work with Netscape 4.0
2. All servers will use a port number between 2100 and 2110 as of November 1.
3. Need for crosswalk for attributes and also implementation guidance documentation.
4. Ask for other examples of tombstone information elements.
5. Poll implementors for work items they would be willing to commit resources, time, and energy to doing as part of an ongoing CIMI Z39.50 Working Group and ongoing CIMI Interoperability Testbed.
Changes to the Profile or Future Profiling Work Items
1. Remove the structure attribute as mandatory (in Appendix C?).
2. IA, p.10: change datatype for resource, add InternationalString.
3. Supporting date ranges and data range searching.
4. Including Aquarelle Profile Use attributes in the CIMI Profile.
5. Museum Object ID needs to be added to the CIMI attribute set.
6. Revise element set names where brief consists of a minimal set of elements sufficient to create a tombstone.
7. Including appropriate appendices to address decisions for including specifications or punting on issues/problems (e.g., SGML). Also, can provide guidance on returning SGML in GRS records and how that could be accomplished, although it was not tested in the interoperability testbed. Another topic would be the relationship of Bib-1 and CIMI.
8. Propose levels of conformance:
9. Where a use attribute is removed, keep the number there but designate it as [reserved], not [undefined].
10. Address the possibility of Who, What, Where, When searches by mapping to existing Use attributes.