User Models for Tailored and Contextual Information Retrieval 

This project is an extension of the work that I did for my dissertation, as part of the Mongrel project at Rutgers.  The Mongrel project (http://www.scils.rutgers.edu/etc/mongrel/) was led by Nick Belkin at Rutgers and was in collaboration with Bruce Croft and James Allan from the University of Massachusetts, Amherst. 

The goals of the current project are to identify, develop, and evaluate techniques for building and maintaining personalized user models, and for using these models to tailor retrieval to individuals.  The project further seeks to understand how the user's information-seeking context can be measured and incorporated into the user model, and how this can be used for contextual retrieval.  Work on this project is ongoing.  Related publications can be found on my home page.  Particular areas of focus:

• Evaluating the efficacy and reliability of using observable behaviors, such as display time and retention, as implicit measures of document preference for use in building these models.  
• Understanding the relationship between the user's task and information-seeking behaviors, and understanding how the user's tasks can be automatically identified by tracking these behaviors over time.
• Understanding the relationship between a user's familiarity with a search topic and the types of documents that the user is likely to be interested in.  What does it mean for a user to be "more" or "less" familiar with a topic?  Is there a difference in the kinds of documents that a user finds relevant based on their level of topic familiarity?  Can levels of familiarity be mapped to particular document features?  Can a user's level of familiarity with a topic be inferred from information-seeking behavior?
• Building a collection of user-annotated web documents.  This collection of documents will include annotated meta-data such as the user's goal in requesting the document, the task that the user was trying to accomplish, the user's level of familiarity with the topic of the document and how useful the user found the document.  This collection will be built by conducting a long-term monitoring study similar to the one in the dissertation. 
• Building valid and reliable tools for collecting and measuring online information-seeking in natural environments.

TREC HARD

This project is part of the National Institute for Standards and Technology's (NIST) Annual Text Retrieval Conference.  The purpose of the HARD track is to "achieve high accuracy retrieval from documents by leveraging additional information about the searcher and/or the search context, through techniques such as passage retrieval, and using very targeted interaction with the searcher" (http://ciir.cs.umass.edu/research/hard/).  I participated in this project last year as part of the Rutgers team with Nick Belkin (before that I participated in the Interactive track for five years), and am leading an effort this year at UNC. 

Our group at UNC is using the lemur toolkit (http://www-2.cs.cmu.edu/~lemur/) for retrieval, and this year, we are investigating techniques for elicting additional information from users (beyond a short query) about their information problems. A general problem in IR is that users typically pose very short queries. It is generally accepted in IR that longer queries result in better retrieval. Thus, we are trying to design a one-shot interaction technique (clarification form) that allows us to get more information from users about their information problems. One of our goals is to make these techniques as least intrusive and disruptive as possible. We are also exploring techniques for incorporating information about the user's search context into retrieval, such as how familiar the user is with their search topic. Finally, as with all TREC projects, we will evaluate the effectiveness of all of the techniques that we develop using standard IR evaluation metrics, and more specifically, precision, since it is highly related to the purposes of the HARD track.

Related publications (from previous TRECs) can be found on my home page.

Student Members of Research Group: Vijay Deepak Dollu and Robert Fu 

Evaluation Metrics for Question-Answering Systems

This project was funded ($500,000+) by ARDA (Advanced Research and Development Activity) as the ARDA Challenge Workshop for 2004. The Co-PIs of this project were Drs. Emile Morse and Jean Scholtz from the National Institute of Standards and Technology. My role was that of Metrics Consultant. This project was a highly intense, fast-paced research effort, which last from May 2004 to August 2004. The goal of the project was to develop evaluation methods and metrics for analytic question-answering (QA) systems. In particular, the research project focused on developing techniques and tools for evaluating interactive QA systems, or QA systems where real users are present. The core of the project was a two-week evaluation workshop of four leading-edge QA systems with eight intelligence analysts. These intelligence analysts came to the evaluation site and conducted searching and other related activities over the course of the two-week time period.

My role as a Metrics Consultant was to design, evaluate, and implement instruments for the workshop, as well as develop and plan the general overall experimental design of the workshop. Instruments included three types of questionnaires, interview schedules, focus group interview schedules, and observation worksheets. In addition to this role, I served as Lead Observer, where I developed training materials on the methods of naturalistic observation and interviewing, and trained other observers on these methods. I also developed an Observer Protocol, a detailed list of step-by-step instructions about how to conduct the experimental sessions. During the workshop, the "observers" (myself included) facilitated the experimental sessions, observered analysts while they searched, and interviewed analysts about their experiences using the different systems. After the workshop ended, I analyzed data from the numerous questionnaires that were adminstered during the workshop, and significantly contributed to the production of the final report.

The outcome of the project was a "gold-standard" methodology, which allowed for the detection of significant differences in the QA systems investigated. This methodology will be used by the research community to conduct future evaluations of QA systems.

Retrieval for Fact- and Task-Oriented Questions

This project is a continuation of a two-year project that I have been working on with Nick Belkin and Xiaojun Yuan from Rutgers, and Vanessa Murdock and Bruce Croft from UMass.  The goals of this project are to (1) develop automatic techniques for classifying web queries that are posed in the form of questions according to fact- or task-orientation; (2) identify features that distinguish documents that are relevant to fact-oriented question from those that are relevant to task-oriented questions; (3) use these differences to tailor retrieval to question type.  [Note: A question like, "Who is the president of the United States?" would be a fact-oriented question, and a question like, "How do I file my income taxes?" would be a task-oriented question.]

Our work has demonstrated that questions can be classified reliably according to orientation and that the features of documents that are relevant to task-oriented questions are different than those that are relevant to fact-oriented questions.   We are currently identifying more useful and distinguishing features, formalizing these differences into retrieval techniques, and evaluating the performance of these techniques.  This work includes conducting studies where users evaluate  retrieved documents for task- and fact-oriented questions.  We expect to finish this project later this year. 

Related publication:  Kelly, D., Murdock, V., Yuan, X.J., Croft, W.B., & Belkin, N.J. (2002). Features of documents relevant to task- and fact-oriented questions. In Proceedings of the Eleventh International Conference on Information and Knowledge Management (CIKM '02), McLean, VA.

Please contact me if you are interested in working on any of these projects.

<back to diane kelly's home page>

Last Updated:  12 August 2004