Diane Kelly - CV

University of North Carolina at Chapel Hill
School of Information and Library Science
204 Manning Hall, CB#3360
Chapel Hill / NC / USA / 27599-3360
TEL: 919.962.8065 / FAX: 919.962.8071
kelly [at] ils.unc.edu

CURRENT APPOINTMENT

Assistant Professor, January 2004 - present
School of Information and Library Science
University of North Carolina at Chapel Hill

EDUCATION

Ph.D., Information Science, January 2004
Rutgers University, New Brunswick, NJ
Committee Members: N. J. Belkin (Advisor), T. Saracevic, D. Riecken, A. Spoerri, & G. Golovchinsky
Dissertation Title: Understanding Implicit Feedback: A Naturalistic User Study

Cognitive Science Certificate, January 2004
Rutgers University Center for Cognitive Science, Rutgers University, Piscataway, NJ
Advisor: Z. Pylyshyn

Master of Library Science, May 1999
School of Communication, Information & Library Studies, Rutgers University, New Brunswick, NJ
Specialization - Information Retrieval

Bachelor of Arts, May 1996
The University of Alabama, Tuscaloosa, AL
Double Major: Psychology & English

FUNDED RESEARCH

An Interface To Support Facet-Based Information Seeking and Retrieval
UNC Junior Faculty Award, $7,500
Principle Investigator, 2006

An Investigation of Evaluation Methods and Metrics for Analytic Question Answering Systems
Advanced Research and Development Activity Challenge Workshop, $500,000
Emile Morse & Jean Scholtz, NIST, Co-PIs
Metrics Consultant, May 2004 - August 2004

OTHER RESEARCH

User Models for Tailored and Contextual Information Retrieval
January 2004 -

UNC TREC (Text Retrieval Conference) - HARD Track (High Accuracy Retrieval of Documents)
Research Team Leader, January 2004 -
Student Members of Team: Robert Fu

High-Quality Interactive Question-Answering (HITIQA)
Graduate Assistant, September 2003 - December 2003
Dr. Paul Kantor, Principal Investigator, SCILS, Rutgers University with
Dr. Tomek Strzalkowski, Principal Investigator, SUNY, Albany

Supporting Effective Access through User- and Topic-Based Language Models
Graduate Assistant, September 2000 - December 2003
Dr. Nicholas J. Belkin, Principal Investigator, SCILS, Rutgers University with 
Dr. W. Bruce Croft and Dr. James Allan, Principal Investigators, CIIR, University of Massachusetts, Amherst

Rutgers TREC (Interactive and HARD Tracks)
Research Team Member, April 1998 - December 2003
Dr. Nicholas J. Belkin, Principal Investigator, SCILS, Rutgers University

Rutgers Distributed Laboratory for Digital Libraries
Research Team Member, March 1999 - 2002
Dr. Paul Kantor, Director, SCILS, Rutgers University

Fuji Xerox Palo Alto Laboratory, Palo Alto, California
Intern, June 2001 - August 2001
Gene Golovchinsky, Sponsor

Rutgers Center for Cognitive Science
Research Assistant, September 1998 - 2001
Dr. Zenon Pylyshyn, Advisor, Department of Cognitive Psychology, Rutgers University

PUBLICATIONS

REFEREED

Book Chapters

Kelly, D. & Teevan, J. (2007). Understanding what works: Evaluating personal information management tools. In W. Jones & J. Teevan (Eds.), Personal Information Management. Seattle: University of Washington Press.

Kelly, D. (2005). Implicit feedback: Using behavior to infer relevance. In A. Spink and C. Cole (Eds.) New Directions in Cognitive Information Retrieval. Springer Publishing: Netherlands (pp.169-186).

Journal Articles

Kelly, D., Kantor, P. B., Morse, E. L., Scholtz, J., & Sun, Y. (to appear). Questionnaires for eliciting evaluation data from users of interactive question answering systems. Journal of Natural Language Engineering (Speical Issue on Interactive Question Answering).

Kelly, D., Harper, D. J., & Landau, B. (2008). Questionnaire mode effects in interactive information retrieval experiments. Information Processing & Management, 44 (1), 122-141.

Wacholder, N., Kelly, D., Kantor, P., Rittman, R., Sun, Y., Bai, B., Small, S., Yamrom, B., & Strzalkowski, T. (2007). A model for realistic evaluation of an end-to-end question answering system. Journal of the American Society for Information Science and Technology 58(8). 1082-1099.

Kelly, D., Wacholder, N., Rittman, R., Sun, Y., Kantor, P., Small, S., & Strzalkowski, T. (2007). Using interview data to identify evaluation criteria for interactive, analytical question answering systems. Journal of the American Society for Information Science and Technology 58(7), 1032-1043.

Murdock, V., Kelly, D., Croft, W. B., Belkin, N. J., & Yuan, X.-J. (2007). Identifying and answering procedural questions. Information Processing & Management 43(1), 181-203.

Kelly, D., & Fu, X. (2007). Eliciting better information need descriptions from users of information systems. Information Processing & Management 43(1), 30-46.

Kelly, D. (2006). Measuring online information-seeking context, part 2. Findings and discussion. Journal of the American Society for Information Science & Technology 57(14), 1862-1874.

Kelly, D. (2006). Measuring online information-seeking context, part 1. Background and method. Journal of the American Society for Information Science & Technology 57(13), 1729-1739.

Belkin, N. J., Cool, C., Kelly, D., Lin, S.-J., Park, S.Y., Perez-Carballo, J., & Sikora, C. (2001). Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval. Information Processing & Management 37(3), 404-434. [pdf]

Conference Papers

White, R. W. & Kelly, D. (2006). A study on the effects of personalization and task information on implicit feedback performance. Conference on Information and Knowledge Management, (CIKM '06).

Harper, D. J. & Kelly, D. (2006). Contextual relevance feedback. Proceedings of the 1st Symposium on Information Interaction in Context (IIiX), Copenhagen, Denmark.

Kelly, D., & Fu, X. (2006). Elicitation of term relevance feedback: An investigation of term source and context. Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '06), Seattle, WA, 453-460.

Kelly, D., Dollu, V. J., & Fu, X. (2005). The loquacious user: A document-independent source of terms for query expansion. In Proceedings of the 28th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '05), Salvador, Brazil, 457-464. [pdf]

Kelly, D. & Belkin, N.J. (2004). Display time as implicit feedback: Understanding task effects. In Proceedings of the 27th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '04), Sheffield, UK, 377-384. [pdf]

Kelly, D., Diaz, F., Belkin, N. J., & Allan, J. (2004). A user-centered approach to evaluating topic models. In Proceedings of the 26th European Conference on Information Retrieval (ECIR '04), Sunderland, UK, 27-41. [pdf]

Small, S., Strzalkowski, T., Lu, T., Ryan, S., Kantor, P., Kelly, D., Rittman, R., & Wacholder, N. (2004). HITIQA: Toward analytical question-answering. Proceedings of the 20th International Conference on Computational Linguistics (COLING '04), Geneva, Switzerland.

Wacholder, N., Small, S., Bai, B., Kelly, D., Rittman, R., Ryan, S., Salkin, R., Song, P., Sun, Y., Ting, L., Kantor, P., & Strazalkowski, T. (2004). Designing a realistic evaluation of an end-to-end interactive question answering system. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC '04), Lisbon, Portugal. [pdf]

Belkin, N. J., Cool, C., Kelly, D., Lee, H.-J., Muresan, G., Tang, M.-C., & Yuan, X.-J. (2003). Query length in interactive information retrieval. In Proceedings of the 26th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '03), Toronto, CA, 205-212. [pdf]

Kelly, D. & Belkin, N. J. (2002). A user modeling system for personalized interaction and tailored retrieval in interactive IR. In Proceedings of Annual Conference of the American Society for Information Science and Technology (ASIST '02), Philadelphia, PA, 316-325. [pdf]

Kelly, D. & Cool, C. (2002). The effects of topic familiarity on information search behavior. In Proceedings of the Second ACM/IEEE Joint Conference on Digital Libraries (JCDL '02), Portland, OR, 74-75. [pdf]

Posters

Fu, X., Kelly, D., & Shah, C. (2007). Using collaborative queries to improve retrieval for difficult topics. Proceedings of the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '07), Amsterdam, The Netherlands, 879-880.

Shah, C., Kelly, D., & Fu, X. (2007). Making mind and machine meet: A study of combining cognitive and algorithmic relevance feedback. Proceedings of the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '07), Amsterdam, The Netherlands, 877-878.

Fu, X., & Kelly, D. (2006). Evaluating sources of query expansion terms. Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '06), Seattle, WA, 647-648.

Kelly, D. (2003). Information search and use behavior as implicit sources of evidence of information problem and document preference. Presented at the Association for Library and Information Science Education (ALISE '03) Annual Conference, Philadelphia, PA.

Kelly, D., Murdock, V., Yuan, X. J., Croft, W. B. & Belkin, N. J. (2002). Features of documents relevant to task- and fact-oriented questions. In Proceedings of the Eleventh International Conference on Information and Knowledge Management (CIKM '02), McLean, VA, 645-647. [best poster award] [pdf]

Kelly, D. & Belkin, N. J. (2001). Reading time, scrolling and interaction: Exploring implicit sources of user preference for relevance feedback. In Proceedings of the 24th Annual International ACM Conference on Research and Development in Information Retrieval (SIGIR '01), New Orleans, LA., 408-409. [pdf]

Workshops

Kelly, D. (2007). Web page relevance: What are we measuring? Workshop on Web Information Seeking and Interaction at the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '07), Amsterdam, The Netherlands.

Kelly, D., Kantor, P. B., Morse, E. L., Scholtz, J., & Sun, Y. (2006). User-centered evaluation of interactive question answering systems. Proceedings of the Workshop on Interactive Question Answering at the Human Language Technology Conference (HLT-NAACL '06), New York, NY.

Kelly, D. (2005). Polyrepresentative extraction of information needs. Presented at the Human Computer Interaction Lab Annual Symposium and Open House, Workshop on Exploratory Search Interfaces, University of Maryland (June 02-03, 2005).

Kelly, D. (2004). Building a test collection for investigating contextual information retrieval. In Proceedings of the 27th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '04), Workshop on Information Retrieval in Context, Sheffield, UK. [Presenter] [pdf]

Small, S., Strzalkowski, T., Lu, T., Ryan, S., Salkin, R., Shimizu, N., Kantor, P., Kelly, D., Rittman, R., Wacholder, N., & Yamrom, B. (2004). HITIQA: Scenario-based question-answering. Proceedings of the Workshop on Pragmatics of Question Answering at the Human Language Technology Conference (HLT-NAACL '04), Boston, MA.

Kelly, D. (2003). Understanding implicit feedback: A naturalistic user study. In Proceedings of the 26th Annual International ACM Conference on Research and Development in Information Retrieval (SIGIR '03) Workshop on Implicit Measures of User Interests and Preferences, Toronto, CA. [Poster]

Kelly, D. & Belkin, N. J. (2002). Modeling characteristics of the user's problematic situation with information search and use behaviors. In Proceedings of the Second ACM/IEEE Joint Conference on Digital Libraries (JCDL '02) Workshop on Document Search Interface Design for Large-scale Collections and Intelligent Access, Portland, OR. [Presenter] [pdf]

Kelly, D. & Belkin, N. J. (2002). Unobtrusively measuring information search and use behaviors using a naturalistic, longitudinal approach. In Proceedings of the Annual Conference of the American Society for Information Science and Technology (ASIST '02) Workshop on Measuring Search Behaviors: Current and Proposed Methods, Philadelphia, PA. [Participant] [pdf]

Tutorials

Kelly, D. (2007). Conducting user-centered information retrieval system evaluations. Proceedings of the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '07), Amsterdam, The Netherlands.

Kelly, D., & Harper, D. J. (2006). Conducting interactive information retrieval user studies. Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '06), Seattle, WA.

OTHER PUBLICATIONS (NON REFEREED)

Kelly, D., & Lin, J. (2007). Overview of the TREC 2006 ciQA Task. SIGIR Forum, 41(1), 107-116.

Kelly, D. (2006). Evaluating personal information management behaviors and tools. Communications of the ACM 49(1), 84-86.

Kelly, D. & Teevan, J. (2003). Implicit feedback for inferring user preference: A bibliography. SIGIR Forum, 37(2), 18-28. [pdf]

TREC (Text Retrieval Conference)

Dang, H., Kelly, D., & Lin, J. (to appear). Overview of the TREC 2007 question answering track. E. Voorhees & L. P. Buckland (Eds.), TREC2007, Proceedings of the Sixteenth Text Retrieval Conference. Washington, D.C.: GPO.

Kelly, D., Murdock, V., & Fu, X. (to appear). Using interactions to improve translation dictionaries: UNC, Yahoo! and ciQA. E. Voorhees & L. P. Buckland (Eds.), TREC2007, Proceedings of the Sixteenth Text Retrieval Conference. Washington, D.C.: GPO.

Dang, H., Lin, J., & Kelly, D. (2007). Overview of the TREC 2006 question answering track. E. Voorhees & L. P. Buckland (Eds.), TREC2006, Proceedings of the Fifteenth Text Retrieval Conference. Washington, D.C.: GPO.

Kelly, D. & Fu, X. (2006). University of North Carolina's HARD Track Experiments at TREC 2005. In E. Voorhees & L. P. Buckland (Eds.), TREC2005, Proceedings of the Fourteenth Text Retrieval Conference. Washington, D.C.: GPO.

Kelly, D., Dollu, V. D., and Fu, X. (2005). University of North Carolina's HARD Track Experiments at TREC 2004. In E. Voorhees & L. P. Buckland (Eds.), TREC2004, Proceedings of the Thirteenth Text Retrieval Conference. Washington, D.C.: GPO.

Belkin, N. J., Kelly, D., Lee, H.-J., Li, Y.-L., Muresan, G., Tang, M.-C., Yuan, X.-J., and Zhang, X.-M. (2004). Rutgers' HARD and Web Interactive Track Experiences at TREC 2003. In E. Voorhees & L. P. Buckland (Eds.), TREC2003, Proceedings of the Twelfth Text Retrieval Conference. Washington, D.C.: GPO.

Belkin, N. J., Cool, C., Kelly, D., Kim, G., Kim, J., Lee, H.-J., Muresan, G., Tang, M.-C., & Yuan, X.-J. (2003). Rutgers' TREC 2002 Interactive Track Experience. In E. Voorhees & L. P. Buckland (Eds.), TREC2002, Proceedings of the Eleventh Text Retrieval Conference. Washington, D.C.: GPO.

Belkin, N. J., Cool, C., Jeng, J., Keller, A., Kelly, D., Kim, J., Lee, H.-J., Tang, M.-C., & Yuan, X.-J. (2002). Rutgers' TREC 2001 Interactive Track Experience. In D. Harman & E. Voorhees (Eds.), TREC2001, Proceedings of the Tenth Text Retrieval Conference. Washington, D.C.: GPO.

Belkin, N. J., Keller, A. M., Kelly, D., Perez-Carballo, J., Sikora, C., & Sun, Y. (2001). Support for question-answering in interactive information seeking: The Rutgers TREC-9 interactive track experience. In D. Harman & E. Voorhees (Eds.), TREC-9, Proceedings of the Ninth Text Retrieval Conference. Washington, D.C.: GPO.

Belkin, N. J., Cool, C., Head, J., Jeng, J., Kelly, D., Lin, S., Park, S. Y., Savage-Knepshield, P. & Sikora, C. (2000). Relevance feedback versus local context analysis as term-suggestion devices: Rutgers' TREC-8 interactive track experience. In D. Harman & E. Voorhees (Eds.), TREC-8, Proceedings of the Eighth Text Retrieval Conference. Washington, D.C.: GPO.

Belkin, N. J., Perez-Carballo, J., Cool, C., Kelly, D., Lin, S., Park, S. Y., Rieh, S. Y., Savage-Knepshield, P. & Sikora, C. (1999). Rutgers' TREC-7 interactive track experience. In D. Harman & E. Voorhees (Eds.), TREC-7, Proceedings of the Seventh Text Retrieval Conference. Washington, D.C.: GPO.

Workshops

The PIM Workshop, NSF-Sponsored Invitational Workshop on Personal Information Management, Seattle, Washington (January 27-29, 2005). [Invited]

Kelly, D. (2003). Workshop on Machine Learning, Information Retrieval, and User Modeling. UM '03, The Ninth International Conference on User Modeling, Johnstown, PA. [Participant]

Dissertation

Kelly, D. (2004). Understanding implicit feedback and document preference: A naturalistic user study (dissertation abstract). SIGIR Forum, 38(1), 77.

Kelly, D. (2004). Understanding implicit feedback and document preference: A naturalistic user study. Ph.D. Dissertation, Rutgers University.

Doctoral Seminars

Kelly, D. (2003). A longitudinal, naturalistic study of information search and use behavior as implicit feedback for user model construction and maintenance. UM '03, The Ninth International Conference on User Modeling, Doctoral Consortium, Johnstown, PA.

Kelly, D. (2002). Information search and use behaviors as implicit sources of evidence of information problem and document preference. In Proceedings of Annual Conference of the American Society for Information Science and Technology (ASIST '02) Doctoral Seminar, Philadelphia, PA.

Kelly, D. (2002). Information search and use behaviors as implicit sources of evidence of information problem and document preference. Presented at the Fourth International Conference on Conceptions of Library and Information Science, (CoLIS '02), Doctoral Forum, Seattle, WA.

Invited Talks

Kelly, D. (2007). What do we want to know? Panel Presentation at the Workshop on Web Information Seeking and Interaction at the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '07), Amsterdam, The Netherlands.

Understanding Implicit Feedback: Behaviors, Context and Relevance. Presentation made at the Association of Library and Information Science Education Conference, January, 2006.

Understanding Implicit Feedback: Behaviors, Context and Relevance. College of Information Studies Research Seminar Series, University of Maryland, November 15, 2005.

Understanding Implicit Feedback: Behaviors, Context and Relevance. Computer Science Department Seminar Series, Virginia Tech, October 07, 2005.

Longitudinal, Naturalistic Study of Information Search and Use Behavior as Implicit Feedback for User Model Construction and Maintenance, Language Technologies Institute (LTI) Seminar Series, Carnegie Mellon University, March 14, 2003.

TEACHING

University of North Carolina

Seminar in Interactive Information Retrieval (INLS 890), Fall 2007
Research Methods (INLS 780): Spring 2004 - Fall 2007 (7 semesters)
Internet Applications (INLS 576): Fall 2004 - Fall 2006 (5 semesters)

Rutgers University

Multimedia Production (17:610:555): Fall 2000, Spring 2001
Internet Interfaces (17:610:559): Summer 2000
Communication Research (04:192:300): Summer 2000
Electronic Commerce (04:192:347): Summer 2000
Library Research: Spring 1999

SERVICE

University of North Carolina

School of Information and Library Science, UNC

Conference Program Committee Member

Workshop Program Committee Member

Journal Editorial Board Member

Journal Paper Reviewer

Editor

Review Panel Member

Professional Affiliations

AWARDS & HONORS 

Graduate

Undergraduate

OTHER WORK EXPERIENCE

Web Developer, Summer 1999
The Women's Center at the County College of Morris, Randolf, NJ

Database Consultant, Summer 1999
Scholarly Communication Center, Alexander Library, Rutgers University, New Brunswick, NJ

Web Developer, January 1999 - September 1999
Dean Kay E. Vandergrift, SCILS, Rutgers University, New Brunswick, NJ

Designer/Lab Assistant, September 1997 - December 1998
Scholarly Communications Center, Alexander Library, Rutgers University, New Brunswick, NJ

Library Assistant, June 1996 - August 1997
Circulation & Reference Departments, Huntsville/Madison County Public Library, Huntsville, AL

Last Updated: July 01, 2007