Diane Kelly Carolina
Tiger
Rescue

My research is focused on information search behavior and the design and evaluation of systems that support interactive information retrieval. I am also interested in experimental design and evaluation measures, and evaluation behavior (that is, how people evaluate search quality).

Publications

Books and Monographs

Ruthven, I. & Kelly, D. (2011). Interactive Information Seeking Behaviour and Retrieval. Facet Press.

Kelly, D. (2009). Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval, 3(1-2), 1-224. DOI: 10.1561/1500000012.

Book Chapters

Kelly, D. & Teevan, J. (2007). Understanding what works: Evaluating personal information management tools. In W. Jones & J. Teevan (Eds.), Personal Information Management. Seattle: University of Washington Press (pp. 190-204).

Kelly, D. (2005). Implicit feedback: Using behavior to infer relevance. In A. Spink and C. Cole (Eds.) New Directions in Cognitive Information Retrieval. Springer Publishing: Netherlands (pp.169-186).

Conference Proceedings (Ed.)

Jones, G. J. F., Sheridan, P., Kelly, D., de Rijke, M. Sakai, T. (2013). Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland.

Belkin, N. J., Kelly, D., Azzopardi, L., Turpin, A. (2010). Proceedings of the 3rd Symposium on Information Interaction in Context (IIiX '10), New Brunswick, NJ.

Shanahan, J. G., Amer-Yahia, S., Zhang, Y., Kolcz, A., Chowdury, A., & Kelly, D. (2008). Proceedings of the 17th ACM Conference on Information and Knowledge Management (CIKM), Napa Valley, CA.

Borlund, P., Schneider, J. W., Lalmas, M., Tombros, A., Feather, J., Kelly, D., de Vries, A., & Azzopardi, L. (2008). Proceedings of the 2nd IIiX Symposium on Information Interaction in Context, London, UK.

Journal Articles

Niu, X. & Kelly, D. (2014). The use of query suggestions as idea tactics during information search. Information Processing & Management, 50(1), 218-234.

Huang, K. & Kelly, D. (2013). The daily image information needs and seeking behaviors of Chinese undergraduate students. College & Research Libraries, 74, 243-261.

Kelly, D. & Sugimoto, C. R. (2013). A systematic review of interactive information retrieval evaluation studies, 1967-2006. Journal of the American Society for Information Science and Technology, 64(4), 745-770.

Kelly, D., Fu, X., & Shah, C. (2010). Effects of rank of relevant documents and search precision on users' evaluations of system performance. ACM Transactions on Information Systems 28(2), Article 9.

Kelly, D., Kantor, P. B., Morse, E. L., Scholtz, J., & Sun, Y. (2009). Questionnaires for eliciting evaluation data from users of interactive question answering systems. Journal of Natural Language Engineering, 15(1), 119-141.

Kelly, D., Harper, D. J., & Landau, B. (2008). Questionnaire mode effects in interactive information retrieval experiments. Information Processing & Management, 44 (1), 122-141.

Wacholder, N., Kelly, D., Kantor, P., Rittman, R., Sun, Y., Bai, B., Small, S., Yamrom, B., & Strzalkowski, T. (2007). A model for realistic evaluation of an end-to-end question answering system. Journal of the American Society for Information Science and Technology 58(8). 1082-1099.

Kelly, D., Wacholder, N., Rittman, R., Sun, Y., Kantor, P., Small, S., & Strzalkowski, T. (2007). Using interview data to identify evaluation criteria for interactive, analytical question answering systems. Journal of the American Society for Information Science and Technology 58(7), 1032-1043.

Murdock, V., Kelly, D., Croft, W. B., Belkin, N. J., & Yuan, X.-J. (2007). Identifying and answering procedural questions. Information Processing & Management 43(1), 181-203.

Kelly, D., & Fu, X. (2007). Eliciting better information need descriptions from users of information systems. Information Processing & Management 43(1), 30-46.

Kelly, D. (2006). Measuring online information-seeking context, part 2. Findings and discussion. Journal of the American Society for Information Science & Technology 57(14), 1862-1874.

Kelly, D. (2006). Measuring online information-seeking context, part 1. Background and method. Journal of the American Society for Information Science & Technology 57(13), 1729-1739.

Belkin, N. J., Cool, C., Kelly, D., Lin, S.-J., Park, S.Y., Perez-Carballo, J., & Sikora, C. (2001). Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval. Information Processing & Management 37(3), 404-434.

Conference Papers

Azzopardi, L., Kelly, D. & Brennan, K. (2013). How query cost affects search behavior. Proceedings of the 36th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '13), Dublin, Ireland, 23-32.

Scholer, F., Kelly, D., Wu, W.-C., Lee, H. & Webber, W. (2013). The effects of threshold priming and Need for Cognition on relevance calibration and assessment. Proceedings of the 36th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '13), Dublin, Ireland, 623-632.

Arguello, J., Wu, W.C., Kelly, D., & Edwards, A. (2012). Task complexity, vertical display and user interaction in aggregated search. Proceedings of the 35th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '12), Portland, OR, 435-444.

Wu, W.C., Kelly, D., & Huang, K. (2012). User evaluation of query quality. Proceedings of the 35th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '12), Portland, OR, 215-224.

Bailey, E., & Kelly, D. (2011). Is amount of effort a better predictor of search success than use of specific search tactics? Proceedings of the American Society for Information Science and Technology Conference, 48(1), 1-10.

Kelly, D. & Gyllstrom, K. (2011). An examination of two delivery modes for interactive search system experiments: Remote and laboratory. Proceedings of the 29th ACM Conference on Human Factors in Computing Systems (CHI '11), Vancouver, Canada.

Hauff, C., Kelly, D., & Azzopardi, L. (2010). A comparison of user and system query performance predictions. Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10), 919-988.

Kelly, D., Cushing, A., Dostert, M., Niu, X., & Gyllstom, K. (2010). Effects of popularity and quality on the usage of query suggestions during information search. Proceeding of the 28th ACM Conference on Human Factors in Computing Systems (CHI '10), Atlanta, GA, 45-54.[honorable mention]

Kelly, D., Gyllstrom, K., & Bailey, E. W. (2009). A comparison of term and query suggestion features for interactive searching. Proceedings of the 32th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '09), Boston, Massachusetts, 371-378.

Shah, C., Marchionini, G., & Kelly, D. (2009). Learning design principles for a collabortive information seeking system. Proceeding of the 27th ACM Conference on Human Factors in Computing Systems (CHI '09) (Extended Abstracts), 3419-3424

Kelly, D., Shah, C., Sugimoto, C. R., Bailey, E. W., Clemens, R. A., Irvine, A. K., Johnson, N. A., Ke, W., Oh, S., Poljakova, A., Rodriguez, M. A., van Noord, M. G., & Zhang, Y. (2008). Effects of performance feedback on users' evaluations of an interactive IR system. Proceedings of the 2nd Symposium on Information Interaction in Context (IIiX), London, UK, 75-82.

White, R. W. & Kelly, D. (2006). A study on the effects of personalization and task information on implicit feedback performance. Conference on Information and Knowledge Management, (CIKM '06).

Harper, D. J. & Kelly, D. (2006). Contextual relevance feedback. Proceedings of the 1st Symposium on Information Interaction in Context (IIiX), Copenhagen, Denmark.

Kelly, D., & Fu, X. (2006). Elicitation of term relevance feedback: An investigation of term source and context. Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '06), Seattle, WA, 453-460.

Kelly, D., Dollu, V. J., & Fu, X. (2005). The loquacious user: A document-independent source of terms for query expansion. In Proceedings of the 28th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '05), Salvador, Brazil, 457-464.

Kelly, D. & Belkin, N.J. (2004). Display time as implicit feedback: Understanding task effects. In Proceedings of the 27th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '04), Sheffield, UK, 377-384.

Kelly, D., Diaz, F., Belkin, N. J., & Allan, J. (2004). A user-centered approach to evaluating topic models. In Proceedings of the 26th European Conference on Information Retrieval (ECIR '04), Sunderland, UK, 27-41.

Small, S., Strzalkowski, T., Lu, T., Ryan, S., Kantor, P., Kelly, D., Rittman, R., & Wacholder, N. (2004). HITIQA: Toward analytical question-answering. Proceedings of the 20th International Conference on Computational Linguistics (COLING '04), Geneva, Switzerland.

Wacholder, N., Small, S., Bai, B., Kelly, D., Rittman, R., Ryan, S., Salkin, R., Song, P., Sun, Y., Ting, L., Kantor, P., & Strazalkowski, T. (2004). Designing a realistic evaluation of an end-to-end interactive question answering system. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC '04), Lisbon, Portugal.

Belkin, N. J., Cool, C., Kelly, D., Lee, H.-J., Muresan, G., Tang, M.-C., & Yuan, X.-J. (2003). Query length in interactive information retrieval. In Proceedings of the 26th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '03), Toronto, CA, 205-212.

Kelly, D. & Belkin, N. J. (2002). A user modeling system for personalized interaction and tailored retrieval in interactive IR. In Proceedings of Annual Conference of the American Society for Information Science and Technology (ASIST '02), Philadelphia, PA, 316-325.

Kelly, D. & Cool, C. (2002). The effects of topic familiarity on information search behavior. In Proceedings of the Second ACM/IEEE Joint Conference on Digital Libraries (JCDL '02), Portland, OR, 74-75.

Poster Papers

Wu, W.C., Kelly, D., Edwards, A., & Arguello, J. (2012). Grannies, tanning beds, tattoos and NASCAR: Evaluation of search tasks with varying levels of cognitive complexity. Proceedings of the 3rd Information Interaction in Context Symposium (IIiX 12), Njjmegen, The Netherlands, 254-257. [best poster award]

Hauff, C., de Jong, F., Kelly, D., & Azzopardi, L. (2010). Query quality: User ratings and system predictions. Proceedings of the 33rd Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '10), Geneva, Switzerland, 743-744.

Bailey, E. W., Kelly, D., & Gyllstrom, K. (2009). Undergraduates' evaluations of assigned search topics. Proceedings of the 32th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '09), Boston, Massachusetts, 812-813.

Dostert, M. & Kelly, D. (2009). Users' stopping behaviors and estimates of recall. Proceedings of the 32th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '09), Boston, Massachusetts, 820-821.

Fu, X., Kelly, D., & Shah, C. (2007). Using collaborative queries to improve retrieval for difficult topics. Proceedings of the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '07), Amsterdam, The Netherlands, 879-880.

Shah, C., Kelly, D., & Fu, X. (2007). Making mind and machine meet: A study of combining cognitive and algorithmic relevance feedback. Proceedings of the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '07), Amsterdam, The Netherlands, 877-878.

Fu, X., & Kelly, D. (2006). Evaluating sources of query expansion terms. Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '06), Seattle, WA, 647-648.

Kelly, D. (2003). Information search and use behavior as implicit sources of evidence of information problem and document preference. Presented at the Association for Library and Information Science Education (ALISE '03) Annual Conference, Philadelphia, PA.

Kelly, D., Murdock, V., Yuan, X. J., Croft, W. B. & Belkin, N. J. (2002). Features of documents relevant to task- and fact-oriented questions. In Proceedings of the Eleventh International Conference on Information and Knowledge Management (CIKM '02), McLean, VA, 645-647. [best poster award]

Kelly, D. & Belkin, N. J. (2001). Reading time, scrolling and interaction: Exploring implicit sources of user preference for relevance feedback. In Proceedings of the 24th Annual International ACM Conference on Research and Development in Information Retrieval (SIGIR '01), New Orleans, LA., 408-409.

Workshop Papers and Posters

Kelly, D. (2009). Query suggestions as idea tactics for information search. HCIR 2009: The Third International Workshop on Human-Computer Interaction and Information Retrieval.

Kelly, D. (2007). Web page relevance: What are we measuring? Workshop on Web Information Seeking and Interaction at the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '07), Amsterdam, The Netherlands.

Kelly, D., Kantor, P. B., Morse, E. L., Scholtz, J., & Sun, Y. (2006). User-centered evaluation of interactive question answering systems. Proceedings of the Workshop on Interactive Question Answering at the Human Language Technology Conference (HLT-NAACL '06), New York, NY.

Kelly, D. (2005). Polyrepresentative extraction of information needs. Presented at the Human Computer Interaction Lab Annual Symposium and Open House, Workshop on Exploratory Search Interfaces, University of Maryland (June 02-03, 2005).

Kelly, D. (2004). Building a test collection for investigating contextual information retrieval. In Proceedings of the 27th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '04), Workshop on Information Retrieval in Context, Sheffield, UK.

Small, S., Strzalkowski, T., Lu, T., Ryan, S., Salkin, R., Shimizu, N., Kantor, P., Kelly, D., Rittman, R., Wacholder, N., & Yamrom, B. (2004). HITIQA: Scenario-based question-answering. Proceedings of the Workshop on Pragmatics of Question Answering at the Human Language Technology Conference (HLT-NAACL '04), Boston, MA.

Kelly, D. (2003). Understanding implicit feedback: A naturalistic user study. In Proceedings of the 26th Annual International ACM Conference on Research and Development in Information Retrieval (SIGIR '03) Workshop on Implicit Measures of User Interests and Preferences, Toronto, CA.

Kelly, D. & Belkin, N. J. (2002). Modeling characteristics of the user's problematic situation with information search and use behaviors. In Proceedings of the Second ACM/IEEE Joint Conference on Digital Libraries (JCDL '02) Workshop on Document Search Interface Design for Large-scale Collections and Intelligent Access, Portland, OR.

Kelly, D. & Belkin, N. J. (2002). Unobtrusively measuring information search and use behaviors using a naturalistic, longitudinal approach. In Proceedings of the Annual Conference of the American Society for Information Science and Technology (ASIST '02) Workshop on Measuring Search Behaviors: Current and Proposed Methods, Philadelphia, PA.

Tutorials

Kelly, D. (2007). Conducting user-centered information retrieval system evaluations. Proceedings of the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '07), Amsterdam, The Netherlands.

Kelly, D., & Harper, D. J. (2006). Conducting interactive information retrieval user studies. Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '06), Seattle, WA.

Panels

Ingwersen, P., Larsen, B., Kelly, D., Wang, P., & Lykke, M. (2010). Perspectives on adaptivity in information retrieval interaction. American Society for Information Science and Technology Conference (ASIST 2010), Pittsburgh, PA.

OTHER PUBLICATIONS (NON REFEREED)

Kelly, D., Arguello, J., & Capra, R. (2013). NSF Workshop on Task-based Information Search Systems. SIGIR Forum, 47(2).

Kelly, D. & Belkin, N. J. (2010). The 3rd Information Interaction in Context Symposium (IIiX '10). SIGIR Forum, 44(2), 13-14.

Kelly, D. & Ruthven, I. (2010). Search procedures revisited. In Larsen, B., Schneider, J.W. & strm, F. (Eds.) The Janus Faced Scholar: A Festschrift in Honor of Peter Ingwersen. Copenhagen: The Royal School of Library and Information Science.

Kelly, D., Dumais, S., & Pedersen, J. (2009). Evaluation challenges and directions for information seeking support systems. IEEE Computer, 42(3), 60-66.

Kelly, D., & Lin, J. (2007). Overview of the TREC 2006 ciQA Task. SIGIR Forum, 41(1), 107-116.

Kelly, D. (2006). Evaluating personal information management behaviors and tools. Communications of the ACM 49(1), 84-86.

Kelly, D. & Teevan, J. (2003). Implicit feedback for inferring user preference: A bibliography. SIGIR Forum, 37(2), 18-28.

TREC (Text Retrieval Conference)

Dang, H., Kelly, D., & Lin, J. (2008). Overview of the TREC 2007 question answering track. E. Voorhees & L. P. Buckland (Eds.), TREC2007, Proceedings of the Sixteenth Text Retrieval Conference. Washington, D.C.: GPO.

Kelly, D., Murdock, V., & Fu, X. (2008). Using interactions to improve translation dictionaries: UNC, Yahoo! and ciQA. E. Voorhees & L. P. Buckland (Eds.), TREC2007, Proceedings of the Sixteenth Text Retrieval Conference. Washington, D.C.: GPO.

Dang, H., Lin, J., & Kelly, D. (2007). Overview of the TREC 2006 question answering track. E. Voorhees & L. P. Buckland (Eds.), TREC2006, Proceedings of the Fifteenth Text Retrieval Conference. Washington, D.C.: GPO.

Kelly, D. & Fu, X. (2006). University of North Carolina's HARD Track Experiments at TREC 2005. In E. Voorhees & L. P. Buckland (Eds.), TREC2005, Proceedings of the Fourteenth Text Retrieval Conference. Washington, D.C.: GPO.

Kelly, D., Dollu, V. D., and Fu, X. (2005). University of North Carolina's HARD Track Experiments at TREC 2004. In E. Voorhees & L. P. Buckland (Eds.), TREC2004, Proceedings of the Thirteenth Text Retrieval Conference. Washington, D.C.: GPO.

Belkin, N. J., Kelly, D., Lee, H.-J., Li, Y.-L., Muresan, G., Tang, M.-C., Yuan, X.-J., and Zhang, X.-M. (2004). Rutgers' HARD and Web Interactive Track Experiences at TREC 2003. In E. Voorhees & L. P. Buckland (Eds.), TREC2003, Proceedings of the Twelfth Text Retrieval Conference. Washington, D.C.: GPO.

Belkin, N. J., Cool, C., Kelly, D., Kim, G., Kim, J., Lee, H.-J., Muresan, G., Tang, M.-C., & Yuan, X.-J. (2003). Rutgers' TREC 2002 Interactive Track Experience. In E. Voorhees & L. P. Buckland (Eds.), TREC2002, Proceedings of the Eleventh Text Retrieval Conference. Washington, D.C.: GPO.

Belkin, N. J., Cool, C., Jeng, J., Keller, A., Kelly, D., Kim, J., Lee, H.-J., Tang, M.-C., & Yuan, X.-J. (2002). Rutgers' TREC 2001 Interactive Track Experience. In D. Harman & E. Voorhees (Eds.), TREC2001, Proceedings of the Tenth Text Retrieval Conference. Washington, D.C.: GPO.

Belkin, N. J., Keller, A. M., Kelly, D., Perez-Carballo, J., Sikora, C., & Sun, Y. (2001). Support for question-answering in interactive information seeking: The Rutgers TREC-9 interactive track experience. In D. Harman & E. Voorhees (Eds.), TREC-9, Proceedings of the Ninth Text Retrieval Conference. Washington, D.C.: GPO.

Belkin, N. J., Cool, C., Head, J., Jeng, J., Kelly, D., Lin, S., Park, S. Y., Savage-Knepshield, P. & Sikora, C. (2000). Relevance feedback versus local context analysis as term-suggestion devices: Rutgers' TREC-8 interactive track experience. In D. Harman & E. Voorhees (Eds.), TREC-8, Proceedings of the Eighth Text Retrieval Conference. Washington, D.C.: GPO.

Belkin, N. J., Perez-Carballo, J., Cool, C., Kelly, D., Lin, S., Park, S. Y., Rieh, S. Y., Savage-Knepshield, P. & Sikora, C. (1999). Rutgers' TREC-7 interactive track experience. In D. Harman & E. Voorhees (Eds.), TREC-7, Proceedings of the Seventh Text Retrieval Conference. Washington, D.C.: GPO.

Dissertation

Kelly, D. (2004). Understanding implicit feedback and document preference: A naturalistic user study (dissertation abstract). SIGIR Forum, 38(1), 77.

Kelly, D. (2004). Understanding implicit feedback and document preference: A naturalistic user study. Ph.D. Dissertation, Rutgers University.

Invited Talks