** Next:** Comparing Retrieval or Search
** Up:** Measuring Search Engine Quality
** Previous:** Experimental Rankings

Given the analytic model of retrieval,
we may compute the *A* values for each query (*A*_{i} represents the *A* value for the
query) and the *Q* for each retrieval engine, where *Q*_{j} represents the quality (probability of optimal ranking) of search engine *j*.
The *A* values may be interpreted as the level of difficulty associated with retrieving the relevant documents on the topic represented by various formulations of the query.
The *Q* values may be interpreted as the quality of each search mechanism.
We compute these values by performing a rather lengthy regression.
Our goal is to solve for the various values of *A*_{i} and *Q*_{j} for each query and each search engine,
finding the set of *A* and *Q* values that minimize the errors made in estimating the ASL values.
This is a complex problem, and there are no standard simple procedures for solving it.
We can treat the problem as being to solve a non-linear regression of the form

Here the ASL is the dependent variable and the parameters
and
are independent variables to be estimated by the regression package.
The variable *x*_{i} is an indicator variable that has the value 1 when the
query in question is query *i*, and 0 otherwise.
The variable *y*_{i} similarly is an indicator variable that has the value
1 when the retrieval engine being used is retrieval engine number *i*, and 0 otherwise.
The data set contains 600 document rankings, one for each combination of the six search techniques and for each of the 100 queries.
The *N* values are set to the correct number of documents for each database.
The numbers that are obtained from these regressions are inexact.
They are estimates that would be better with a larger sample of queries and documents from which to make the estimates.
The standard errors for estimating *Q* values are all approximately 0.014, while the standard errors for estimating *A* values are approximately 0.056.
The *Q* values reflect the database from which they are derived.
The *A* values are query specific and reflect the nature of the relevance judgments and the documents available.
The *Q* values are computed so as to mathematically complement the *A* values so the regression formula produces an ASL values with minimal error.
While *Q* values clearly will vary due to the characteristics of a specific database, the variance should be relatively small compared to the variation obtained with other measures of retrieval performance quality, such as precision.
In the following section we examine the *Q* values and their robustness.

** Next:** Comparing Retrieval or Search
** Up:** Measuring Search Engine Quality
** Previous:** Experimental Rankings
*Bob Losee*

*1999-07-29*