General Interests
I am interested in the design and evaluation of systems that support interactive information retrieval. My research explores techniques that support interactive information retrieval at the search interface, such as those used for explicit and implicit relevance feedback. My research also focuses on user modeling and personalization. Specifically, I am interested in identifying and evaluating how an online information system can learn and use the document preferences of its users through monitoring observable behaviors, such as display time and retention. I am also interested in understanding how contextual variables, such as specific task and topic, affect this relationship, and how these variables can be measured and tracked over time.
Areas of Interest
explicit and implicit relevance feedback, personalization, information-seeking behavior, user modeling, interactive information retrieval, interfaces for information retrieval, evaluation metrics, experimental design and analysis, quantitative and qualitative research methods.
Publications
REFEREED
Book Chapters
Kelly, D. & Teevan, J. (2007). Understanding what works: Evaluating personal information management tools. In W. Jones & J. Teevan (Eds.), Personal Information Management. Seattle: University of Washington Press.
Kelly, D. (2005). Implicit feedback: Using behavior to infer relevance. In A. Spink and C. Cole (Eds.) New Directions in Cognitive Information Retrieval. Springer Publishing: Netherlands (pp.169-186).
Journal Articles
Kelly, D., Kantor, P. B., Morse, E. L., Scholtz, J., & Sun, Y. (to appear). Questionnaires for eliciting evaluation data from users of interactive question answering systems. Journal of Natural Language Engineering (Speical Issue on Interactive Question Answering).
Kelly, D., Harper, D. J., & Landau, B. (2008). Questionnaire mode effects in interactive information retrieval experiments. Information Processing & Management, 44 (1), 122-141.
Wacholder, N., Kelly, D., Kantor, P., Rittman, R., Sun, Y., Bai, B., Small, S., Yamrom, B., & Strzalkowski, T. (2007). A model for realistic evaluation of an end-to-end question answering system. Journal of the American Society for Information Science and Technology 58(8). 1082-1099.
Kelly, D., Wacholder, N., Rittman, R., Sun, Y., Kantor, P., Small, S., & Strzalkowski, T. (2007). Using interview data to identify evaluation criteria for interactive, analytical question answering systems. Journal of the American Society for Information Science and Technology 58(7), 1032-1043.
Murdock, V., Kelly, D., Croft, W. B., Belkin, N. J., & Yuan, X.-J. (2007). Identifying and answering procedural questions. Information Processing & Management 43(1), 181-203.
Kelly, D., & Fu, X. (2007). Eliciting better information need descriptions from users of information systems. Information Processing & Management 43(1), 30-46.
Kelly, D. (2006). Measuring online information-seeking context, part 2. Findings and discussion. Journal of the American Society for Information Science & Technology 57(14), 1862-1874.
Kelly, D. (2006). Measuring online information-seeking context, part 1. Background and method. Journal of the American Society for Information Science & Technology 57(13), 1729-1739.
Belkin, N. J., Cool, C., Kelly, D., Lin, S.-J., Park, S.Y., Perez-Carballo, J., & Sikora, C. (2001). Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval. Information Processing & Management 37(3), 404-434. [pdf]
Conference PapersWhite, R. W. & Kelly, D. (2006). A study on the effects of personalization and task information on implicit feedback performance. Conference on Information and Knowledge Management, (CIKM '06).
Harper, D. J. & Kelly, D. (2006). Contextual relevance feedback. Proceedings of the 1st Symposium on Information Interaction in Context (IIiX), Copenhagen, Denmark.
Kelly, D., & Fu, X. (2006). Elicitation of term relevance feedback: An investigation of term source and context. Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '06), Seattle, WA, 453-460.
Kelly, D., Dollu, V. J., & Fu, X. (2005). The loquacious user: A document-independent source of terms for query expansion. In Proceedings of the 28th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '05), Salvador, Brazil, 457-464. [pdf]
Kelly, D. & Belkin, N.J. (2004). Display time as implicit feedback: Understanding task effects. In Proceedings of the 27th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '04), Sheffield, UK, 377-384. [pdf]
Kelly, D., Diaz, F., Belkin, N. J., & Allan, J. (2004). A user-centered approach to evaluating topic models. In Proceedings of the 26th European Conference on Information Retrieval (ECIR '04), Sunderland, UK, 27-41. [pdf]
Small, S., Strzalkowski, T., Lu, T., Ryan, S., Kantor, P., Kelly, D., Rittman, R., & Wacholder, N. (2004). HITIQA: Toward analytical question-answering. Proceedings of the 20th International Conference on Computational Linguistics (COLING '04), Geneva, Switzerland.
Wacholder, N., Small, S., Bai, B., Kelly, D., Rittman, R., Ryan, S., Salkin, R., Song, P., Sun, Y., Ting, L., Kantor, P., & Strazalkowski, T. (2004). Designing a realistic evaluation of an end-to-end interactive question answering system. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC '04), Lisbon, Portugal. [pdf]
Belkin, N. J., Cool, C., Kelly, D., Lee, H.-J., Muresan, G., Tang, M.-C., & Yuan, X.-J. (2003). Query length in interactive information retrieval. In Proceedings of the 26th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '03), Toronto, CA, 205-212. [pdf]
Kelly, D. & Belkin, N. J. (2002). A user modeling system for personalized interaction and tailored retrieval in interactive IR. In Proceedings of Annual Conference of the American Society for Information Science and Technology (ASIST '02), Philadelphia, PA, 316-325. [pdf]
Kelly, D. & Cool, C. (2002). The effects of topic familiarity on information search behavior. In Proceedings of the Second ACM/IEEE Joint Conference on Digital Libraries (JCDL '02), Portland, OR, 74-75. [pdf]
Posters
Fu, X., Kelly, D., & Shah, C. (2007). Using collaborative queries to improve retrieval for difficult topics. Proceedings of the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '07), Amsterdam, The Netherlands, 879-880.
Shah, C., Kelly, D., & Fu, X. (2007). Making mind and machine meet: A study of combining cognitive and algorithmic relevance feedback. Proceedings of the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '07), Amsterdam, The Netherlands, 877-878.
Fu, X., & Kelly, D. (2006). Evaluating sources of query expansion terms. Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '06), Seattle, WA, 647-648.
Kelly, D. (2003). Information search and use behavior as implicit sources of evidence of information problem and document preference. Presented at the Association for Library and Information Science Education (ALISE '03) Annual Conference, Philadelphia, PA.
Kelly, D., Murdock, V., Yuan, X. J., Croft, W. B. & Belkin, N. J. (2002). Features of documents relevant to task- and fact-oriented questions. In Proceedings of the Eleventh International Conference on Information and Knowledge Management (CIKM '02), McLean, VA, 645-647. [best poster award] [pdf]
Kelly, D. & Belkin, N. J. (2001). Reading time, scrolling and interaction: Exploring implicit sources of user preference for relevance feedback. In Proceedings of the 24th Annual International ACM Conference on Research and Development in Information Retrieval (SIGIR '01), New Orleans, LA., 408-409. [pdf]
Workshops
Kelly, D. (2007). Web page relevance: What are we measuring? Workshop on Web Information Seeking and Interaction at the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '07), Amsterdam, The Netherlands.
Kelly, D., Kantor, P. B., Morse, E. L., Scholtz, J., & Sun, Y. (2006). User-centered evaluation of interactive question answering systems. Proceedings of the Workshop on Interactive Question Answering at the Human Language Technology Conference (HLT-NAACL '06), New York, NY.
Kelly, D. (2005). Polyrepresentative extraction of information needs. Presented at the Human Computer Interaction Lab Annual Symposium and Open House, Workshop on Exploratory Search Interfaces, University of Maryland (June 02-03, 2005).
Kelly, D. (2004). Building a test collection for investigating contextual information retrieval. In Proceedings of the 27th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '04), Workshop on Information Retrieval in Context, Sheffield, UK. [Presenter] [pdf]
Small, S., Strzalkowski, T., Lu, T., Ryan, S., Salkin, R., Shimizu, N., Kantor, P., Kelly, D., Rittman, R., Wacholder, N., & Yamrom, B. (2004). HITIQA: Scenario-based question-answering. Proceedings of the Workshop on Pragmatics of Question Answering at the Human Language Technology Conference (HLT-NAACL '04), Boston, MA.
Kelly, D. (2003). Understanding implicit feedback: A naturalistic user study. In Proceedings of the 26th Annual International ACM Conference on Research and Development in Information Retrieval (SIGIR '03) Workshop on Implicit Measures of User Interests and Preferences, Toronto, CA. [Poster]
Kelly, D. & Belkin, N. J. (2002). Modeling characteristics of the user's problematic situation with information search and use behaviors. In Proceedings of the Second ACM/IEEE Joint Conference on Digital Libraries (JCDL '02) Workshop on Document Search Interface Design for Large-scale Collections and Intelligent Access, Portland, OR. [Presenter] [pdf]
Kelly, D. & Belkin, N. J. (2002). Unobtrusively measuring information search and use behaviors using a naturalistic, longitudinal approach. In Proceedings of the Annual Conference of the American Society for Information Science and Technology (ASIST '02) Workshop on Measuring Search Behaviors: Current and Proposed Methods, Philadelphia, PA. [Participant] [pdf]
Tutorials
Kelly, D. (2007). Conducting user-centered information retrieval system evaluations. Proceedings of the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '07), Amsterdam, The Netherlands.
Kelly, D., & Harper, D. J. (2006). Conducting interactive information retrieval user studies. Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '06), Seattle, WA.
OTHER PUBLICATIONS (NON REFEREED)
Kelly, D., & Lin, J. (2007). Overview of the TREC 2006 ciQA Task. SIGIR Forum, 41(1), 107-116.
Kelly, D. (2006). Evaluating personal information management behaviors and tools. Communications of the ACM 49(1), 84-86.
Kelly, D. & Teevan, J. (2003). Implicit feedback for inferring user preference: A bibliography. SIGIR Forum, 37(2), 18-28. [pdf]
TREC (Text Retrieval Conference)
Dang, H., Kelly, D., & Lin, J. (to appear). Overview of the TREC 2007 question answering track. E. Voorhees & L. P. Buckland (Eds.), TREC2007, Proceedings of the Sixteenth Text Retrieval Conference. Washington, D.C.: GPO.
Kelly, D., Murdock, V., & Fu, X. (to appear). Using interactions to improve translation dictionaries: UNC, Yahoo! and ciQA. E. Voorhees & L. P. Buckland (Eds.), TREC2007, Proceedings of the Sixteenth Text Retrieval Conference. Washington, D.C.: GPO.
Dang, H., Lin, J., & Kelly, D. (2007). Overview of the TREC 2006 question answering track. E. Voorhees & L. P. Buckland (Eds.), TREC2006, Proceedings of the Fifteenth Text Retrieval Conference. Washington, D.C.: GPO.
Kelly, D. & Fu, X. (2006). University of North Carolina's HARD Track Experiments at TREC 2005. In E. Voorhees & L. P. Buckland (Eds.), TREC2005, Proceedings of the Fourteenth Text Retrieval Conference. Washington, D.C.: GPO.
Kelly, D., Dollu, V. D., and Fu, X. (2005). University of North Carolina's HARD Track Experiments at TREC 2004. In E. Voorhees & L. P. Buckland (Eds.), TREC2004, Proceedings of the Thirteenth Text Retrieval Conference. Washington, D.C.: GPO.
Belkin, N. J., Kelly, D., Lee, H.-J., Li, Y.-L., Muresan, G., Tang, M.-C., Yuan, X.-J., and Zhang, X.-M. (2004). Rutgers' HARD and Web Interactive Track Experiences at TREC 2003. In E. Voorhees & L. P. Buckland (Eds.), TREC2003, Proceedings of the Twelfth Text Retrieval Conference. Washington, D.C.: GPO.
Belkin, N. J., Cool, C., Kelly, D., Kim, G., Kim, J., Lee, H.-J., Muresan, G., Tang, M.-C., & Yuan, X.-J. (2003). Rutgers' TREC 2002 Interactive Track Experience. In E. Voorhees & L. P. Buckland (Eds.), TREC2002, Proceedings of the Eleventh Text Retrieval Conference. Washington, D.C.: GPO.
Belkin, N. J., Cool, C., Jeng, J., Keller, A., Kelly, D., Kim, J., Lee, H.-J., Tang, M.-C., & Yuan, X.-J. (2002). Rutgers' TREC 2001 Interactive Track Experience. In D. Harman & E. Voorhees (Eds.), TREC2001, Proceedings of the Tenth Text Retrieval Conference. Washington, D.C.: GPO.
Belkin, N. J., Keller, A. M., Kelly, D., Perez-Carballo, J., Sikora, C., & Sun, Y. (2001). Support for question-answering in interactive information seeking: The Rutgers TREC-9 interactive track experience. In D. Harman & E. Voorhees (Eds.), TREC-9, Proceedings of the Ninth Text Retrieval Conference. Washington, D.C.: GPO.
Belkin, N. J., Cool, C., Head, J., Jeng, J., Kelly, D., Lin, S., Park, S. Y., Savage-Knepshield, P. & Sikora, C. (2000). Relevance feedback versus local context analysis as term-suggestion devices: Rutgers' TREC-8 interactive track experience. In D. Harman & E. Voorhees (Eds.), TREC-8, Proceedings of the Eighth Text Retrieval Conference. Washington, D.C.: GPO.
Belkin, N. J., Perez-Carballo, J., Cool, C., Kelly, D., Lin, S., Park, S. Y., Rieh, S. Y., Savage-Knepshield, P. & Sikora, C. (1999). Rutgers' TREC-7 interactive track experience. In D. Harman & E. Voorhees (Eds.), TREC-7, Proceedings of the Seventh Text Retrieval Conference. Washington, D.C.: GPO.
Workshops
The PIM Workshop, NSF-Sponsored Invitational Workshop on Personal Information Management, Seattle, Washington (January 27-29, 2005). [Invited]
Kelly, D. (2003). Workshop on Machine Learning, Information Retrieval, and User Modeling. UM '03, The Ninth International Conference on User Modeling, Johnstown, PA. [Participant]
Dissertation
Kelly, D. (2004). Understanding implicit feedback and document preference: A naturalistic user study (dissertation abstract). SIGIR Forum, 38(1), 77.
Kelly, D. (2004). Understanding implicit feedback and document preference: A naturalistic user study. Ph.D. Dissertation, Rutgers University.
Doctoral Seminars
Kelly, D. (2003). A longitudinal, naturalistic study of information search and use behavior as implicit feedback for user model construction and maintenance. UM '03, The Ninth International Conference on User Modeling, Doctoral Consortium, Johnstown, PA.
Kelly, D. (2002). Information search and use behaviors as implicit sources of evidence of information problem and document preference. In Proceedings of Annual Conference of the American Society for Information Science and Technology (ASIST '02) Doctoral Seminar, Philadelphia, PA.
Kelly, D. (2002). Information search and use behaviors as implicit sources of evidence of information problem and document preference. Presented at the Fourth International Conference on Conceptions of Library and Information Science, (CoLIS '02), Doctoral Forum, Seattle, WA.
Invited Talks
Kelly, D. (2007). What do we want to know? Panel Presentation at the Workshop on Web Information Seeking and Interaction at the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '07), Amsterdam, The Netherlands.
Understanding Implicit Feedback: Behaviors, Context and Relevance. Presentation made at the Association of Library and Information Science Education Conference, January, 2006.
Understanding Implicit Feedback: Behaviors, Context and Relevance. College of Information Studies Research Seminar Series, University of Maryland, November 15, 2005.
Understanding Implicit Feedback: Behaviors, Context and Relevance. Computer Science Department Seminar Series, Virginia Tech, October 07, 2005.
Longitudinal, Naturalistic Study of Information Search and Use Behavior as Implicit Feedback for User Model Construction and Maintenance, Language Technologies Institute (LTI) Seminar Series, Carnegie Mellon University, March 14, 2003.