next up previous
Next: System Quality and Query Up: Document Rankings from Commercial Previous: Document Rankings from Commercial

Experimental Rankings

A series of document rankings were obtained and then analyzed to determine both the retrieval performance of different search mechanisms and the difficulty or quality of individual queries and topics. The document rankings were from the subsets of documents on the appropriate system that contained the medical subject heading CYSTIC FIBROSIS (CF) in the MEDLINE database during the period from 1974 to 1979 [SWWT91,Par98,PT98]. The original CF queries were produced by subject specialists and are described in Shaw et al. [SWWT91]. Different forms for the queries have been produced by Tibbo, as described in Paris and Tibbo [PT98]. Exhaustive relevance judgments have been obtained for this data, making it an attractive subset of MEDLINE for studies of retrieval performance. Paris [Par98] developed the six sets of document rankings referred to below as Boolean, Freestyle1, Freestyle2, Freestyle3, Target1 and Target2. The first, referred to below as Boolean, represents the retrieval performance obtained with a set of Boolean queries developed by Helen Tibbo for her work with the CF database. In her study, Tibbo found which of several forms for a query produced the best results, and this form of the query is used in this study. Freestyle1 and Target1 represent document rankings produced by searches on the corresponding system using terms from the original natural language queries. Freestyle2 and Target2 represent queries constructed from the terms used in the optimal queries developed by Helen Tibbo. For Freestyle2 and Target2, terms were placed in a single set of quotes if they were linked using the adjacency operator in the optimal Boolean query, e.g. information ADJ retrieval would become ``information retrieval." Terms connected by an OR were placed in parentheses, which are used to approximate OR in Target and Freestyle. Freestyle3 contains the full natural language form of the query, e.g., ``What are the hepatic complications or manifestations of CF?" Minor changes that had to be made in a few specific cases due to system limitations are described in Paris [Par98]. Of greatest interest here is that Freestyle rankings are limited to 1000 documents, while those of the Boolean system includes all 1239 documents in the CF database. Target retrieves up to 50 documents. Note that our examination of these commercial search engines, as well as some earlier studies, are based on retrieval using titles, abstracts, and the sophisticated controlled vocabulary used by the National Library of Medicine. Those search engines using techniques optimized for full-text retrieval will perform somewhat differently with entire documents than they do with the CF database.
next up previous
Next: System Quality and Query Up: Document Rankings from Commercial Previous: Document Rankings from Commercial
Bob Losee
1999-07-29