School of Information and Library Science
University of North Carolina at Chapel Hill

INLS 782, Information Systems Effectiveness
Fall 2006

Class Schedule and Reading List

Bookmark for today

Syllabus / Assignments / Roster / Discussion Forum (INLS782-001, 782)

Table of Contents

Introduction to evaluation: August 24-31
Measurement issues: September 7-28
Designing an evaluation study: October 5-19
Analyzing and interpreting the data: October 26 - November 26
Drawing conclusions from the results: November 30 - December 12

Introduction to Evaluation

1, August 24: Scope of course; Discussion and selection of case studies; Defining system/service objectives

• Stylianou, A. C., & Kumar, R. L. (2000). An integrative framework for IS quality management. Communications of the ACM, 43(9), 99-104. [ACM Digital Library]
• Eisenmann, T. R. (2006). Yahoo! Messenger: Network integration. Harvard Business School, 9-805-102.
• McAfee, A. P., & Sjöman, A. (2006). Wikis at Dresdner Kleinwort Wasserstein: (A). Harvard Business School, 9-606-074.
• McAfee, A., & Wagonfeld, A. B. (2004). Business intelligence software at SYSCO. Harvard Business School, 9-604-080.
• Tufano, P., & Schneider, D. (2005). E-duction, Inc. Harvard Business School, 9-206-006.

• Background reading on instant messaging services
  • Nardi, B.A., Whittaker, S., & Bradner, E. (2000). Interaction and outeraction: Instant messaging in action. Proceedings of CSCW '00, 79-88. [ACM Digital Library]
  • Isaacs, E., et al. (2002). The character, functions, and styles of instant messaging in the workplace. Proceedings of CSCW '02, 11-20. [ACM Digital Library]
  • Grinter, R.E., & Palen, L. (2002). Instant messaging in teen life. Proceedings of CSCW '02, 21-30. [ACM Digital Library]
  • Tyson, J. (n.d.) How instant messaging works. In HowStuffWorks. http://www.howstuffworks.com/instant-messaging.htm.
  • Instant messaging. (2006). In Wikipedia. http://en.wikipedia.org/wiki/Instant_messaging.
• Background reading on wikis
  • Wiki. (2006). In Wikipedia. http://en.wikipedia.org/wiki/Wiki.
  • Tonkin, E. (2005). Making the case for a wiki. Ariadne, Issue 42. http://www.ariadne.ac.uk/issue42/tonkin/.
  • Gonzalez-Reinhart, J. (2005). Wiki and the wiki way: Beyond a knowledge management solution. INformation Systems Research Center, Bauer College of Business, University of Houston. http://www.uhisrc.com/FTB/Wiki/wiki_way_brief%255B1%255D-Jennifer%252005.pdf.
  • Totty, M. (2004, Jan 12). Technology (a special report): Business solutions. Wall Street Journal (Eastern Edition), R4. [Available via ProQuest]
  • Brain, M. (n.d.) How wikis work. In HowStuffWorks. http://computer.howstuffworks.com/wiki.htm.
• Background reading on business intelligence systems
  • Negash, S., & Gray, P. (2003). Business intelligence. Proceedings of the Ninth Americas Conference on Information Systems, 3190-3199. http://www.terry.uga.edu/~jaronson/DSS-Readings/GrayAMCIS2003AugBITutorial.pdf.
  • Thierauf, R. J. (2001). I.1, Introduction to effective business intelligence systems. In Effective Business Intelligence Systems. Westport, CT: Quorum Books, 3-34. [Available via NetLibrary, UNC libraries]
  • Dragoon, A. (2003, Sep 15). Business intelligence gets smart(er). CIO Magazine. http://www.cio.com/archive/091503/smart.html.
• Background reading on payroll deduction and credit
  • Anguelov, C. E., Hilgert, M.A., & Hogarth, J.M. (2004). U.S. consumers and electronic banking. Federal Reserve Bulletin. http://www.federalreserve.gov/pubs/bulletin/2004/winter04_ca.pdf.
  • E-duction, Inc. website. http://www.e-duction.com/.
  • The new wave of plastic perks. (2001, Dec 27). The Motley Fool. http://www.fool.com/news/take/2001/take011227.htm#the.
  • Payroll debit cards score big with banked employees. (2006, Mar 1). eCount. http://www.ecount.com/articles/2006-03-01-release.html.
  • Soman, D. (2003). The effect of payment transparency on consumption: Quasi-experiments from the field. Marketing Letters, 14(3), 173-183. (Focus on third study) [Available via UNC libraries]
  • Barr, M. S. (In press). Banking the poor. Yale Journal on Regulation. http://www.assetbuilding.org/AssetBuilding/Download_Docs/Doc_File_698_1.pdf.

2, August 31: The evaluation process; Establishing evaluation criteria

• Frechtling, J. (2002). The 2002 User Friendly Handbook for Project Evaluation. National Science Foundation. http://www.nsf.gov/pubs/2002/nsf02057/nsf02057.pdf or http://www.nsf.gov/pubs/2002/nsf02057/start.htm
  • Evaluation and types of evaluation, p3-14
• Trochim, W. M. (2005). The Research Methods Knowledge Base. 2nd Edition. http://trochim.human.cornell.edu/kb/index.htm
  • The planning-evaluation cycle, http://www.socialresearchmethods.net/kb/pecycle.htm
• DeLone, W. H., & McLean, E. R. (1992). Information systems success: the quest for the dependent variable. Information Systems Research, 3(1), 60-95. [Available via EBSCOhost]
• Ward, J. A. (1996). Measurement management: what you measure is what you get. Information Systems Management, 13(1), 59-61. [Available via EBSCOhost, html only]

• Begin working on measurement memos
• Connect to the ISWorld "systems effectiveness" page and the ISWorld page on "Survey Instruments in IS"; search the TestLink database from Educational Testing Service's Test Collection; consult the Mental Measurements Yearbook [Davis reference - Z5814.P8 B932]; search the literature on your topic

Measurement Issues

3, September 7: Introduction to measurement issues, with examples

• Sethi, V., & King, W. R. (1991). Construct measurement in information systems research: An illustration in strategic systems. Decision Sciences, 22(3), 455-472. [available from instructor]
• van der Heijden, H., & Verhagen, T. (2004). Online store image: Conceptual foundations and empirical measurement. Information & Management, 41(5), 609-617. [Available via UNC libraries]
• Hernon, P., & Calvert, P. (2005). E-service quality in libraries: Exploring its features and dimensions. Library & Information Science Research, 27(3), 377-404. [Available via UNC libraries]

4, September 14: System/software quality; Information/data quality

• Grady, R.B. (1994). Successfully applying software metrics. IEEE Computer, 27(9), 18-25. [Available via UNC libraries]
• Edberg, D. T. (1997). Creating a balanced IS measurement program. Information Systems Management, 14(2), 32-40. [Available online via EBSCOhost, html only]
• Pipino, L.L., Lee, Y.W., & Wang, R. Y. (2002). Data quality assessment. Communications of the ACM, 45(4 virtual extension), 211-218. [ACM Digital Library]
• Strong, D. M., Lee, Y. W., & Wang, R. Y. (1997). 10 potholes in the road to information quality. Computer, 30(8), 38-46. [Available via UNC libraries]
• Chae, M., Kim, J., Kim, H., & Ryu, H. (2002). Information quality for mobile internet services: a theoretical model with empirical validation. Electronic Markets, 12(1), 38-46. (Focus on p.38-40; assigned by Marcos Rodriguez.) [Available via EBSCOhost]
• Wixom, B.H., & Watson, H.J. (2001). An empirical investigation of the factors affecting data warehousing success. MIS Quarterly, 25(1), 17-41. (Focus on p.19-21, The Research Model and Information Systems Success; Assigned by Mark Pope.) [Available via EBSCOhost]
• Pitt, L.F., Watson, R.T., & Kavan, C.B. (1995). Service quality: A measure of information systems effectiveness. MIS Quarterly, 19(2), 173-187. (Focus on p.177; Assigned by Gershom Rogers) [Available via EBSCOhost]

• Measurement memos on system/software quality and information/data quality due

5, September 21: Usability/performance; Attitudes/satisfaction/user experience

• Rubin, Jeffrey. (1994). Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. New York: John Wiley & Sons. [SILS reserve - QA76.9 .U83 R82 1994]
  • Chapter 2, Overview of usability testing, pages 25-46
• Thong, J. Y. L., & Yap, C. (1996). Information systems effectiveness: A user satisfaction approach. Information Processing & Management, 32(5), 601-610. [Available via UNC libraries]
• Melone, N.P. (1990). A theoretical assessment of the user-satisfaction construct in information systems research. Management Science, 36(1), 76-91. (Focus on sections 1, 2, 3, and the first part of 4.) [Available via EBSCOhost]
• De Wulf, K., Schillewaert, N., Muylle, S., & Rangarajan, D. (2006). The role of pleasure in web site success. Information & Management, 43(4), 434-446. (Focus on sections 1-4, p434-438. [Available via UNC libraries]
• Désilets, A., Paquet, S., & Vinson, N.G. (2005). Are wikis usable? Proceedings of the 2005 International Symposium on Wikis, 3-15. (Focus on the intro to section 3.1, section 3.1.2, and section 3.2; Assigned by Marcos Rodriguez, usability/performance) [Available via the ACM Digital Library]
• Danehower, C., & Lust, J.A. (1995). Understanding and measuring employee benefit satisfaction. Benefits Quarterly, 11(1), 69-75. (Focus on pages 70-72; Assigned by Susan Rathbun-Grubb) [Available via EBSCOhost]
• Ong, C.-S., & Lai, J.-Y. (2004). Developing an instrument for measuring user satisfaction with knowledge management systems. Proceedings of the 37th Annual Hawaii International Conference on System Sciences, 262-271. (Read the entire article, with focus on the first 2 sections; Assigned by Jesse Safir) [Available via IEEE Explore]

• Measurement memos on usability/performance and attitudes/satisfaction due

6, September 28: Adoption/implementation/use; Effects on user effectiveness/productivity; Effects on organizational performance/productivity (notes)

• Trice, A. W., & Treacy, M. E. (1988). Utilization as a dependent variable in MIS research. Data Base, 19(3/4), 33-41. [ACM Digital Library]
• Diffusion theory: Rogers, E. (1995). Diffusion of Innovations. 4th ed. New York: Free Press. [SILS reserve- HM101 .R57 1995]
  • Optional: Chapter 1, Elements of diffusion, p1-37
  • Chapter 5, The innovation-decision process, p161-203 (Focus on p162-186)
• Technology Acceptance Model: Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425-478. (Focus on Tables 1-3 and 9-13.) [Available via EBSCOhost]
• Kohli, R., & Sherer, S. A. (2002). Measuring payoff of information technology investments: Research issues and guidelines. Communications of the AIS, 9, 241-268. (Focus on p241-244.) [Available via UNC libraries]
• Ryker, R., & Nath, R. (1995). An empirical examination of the impact of computer information systems on users. Information & Management, 19, 207-214. (Read Sections 1 & 2 and Appendix A; focus on job satisfaction; assigned by Gershom Rogers) [Available via UNC libraries]
• O'Neill, B., Sorhaindo, B., Prawitz, A., Kim, J., & Garman, E.T. (2006). Financial distress: Definition, effects and measurement. Consumer Interests Annual: Proceedings of the 52nd Annual Conference of the American Council on Consumer Interests, 52, 476-483. (Read entire article; assigned by Susan Rathbun-Grubb) [http://www.consumerinterests.org/i4a/pages/Index.cfm?pageid=4208]
• Leidner, D.E., & Elam, J.J. (1995). The imact of executive information systems on organizational design, intelligence, and decision making. Organization Science, 6(6), 645-664. (Focus on information availability and problem identification/decision making; see especially Figure 1 and pages 649-652; assigned by Mark Pope) [Available via EBSCOhost]
• Devaraj, S., & Kohli, R. (2003). Performance impacts of information technology: Is actual usage the missing link? Management Science, 49(3), 273-289. (Read Sections 1, 2, and 3.4; focus on both use and organizational impact; assigned by Meredith Weiss) [Available via EBSCOhost]
• Majchrzak, A., Wagner, C., & Yates, D. (2006). Corporate wiki users: Results of a survey. Proceedings of the 2006 International Symposium on Wikis, 99-104. (Skim the entire article; focus on section 3.2 and the final part of Table 1; assigned by Jesse Safir) [Available via the ACM Digital Library]
• Optional: Goodhue, D. L., & Thompson, R. L. (1995). Task-technology fit and individual performance. MIS Quarterly, 19(2): 213-236. (Read pages 213-224.) [Available via EBSCOhost]
• Optional: Hitt, L. M., & Brynjolfsson, E. (1996). Productivity, business profitability, and consumer surplus: Three different measures of information technology value. MIS Quarterly, 20(2), 121-142. (Read the first two sections, pages 121-127.) [Available via EBSCOhost]
• Optional: Irani, Z. (2002). Information systems evaluation: navigating through the problem domain. Information & Management, 40(1), 11-24. [Available online via UNC libraries]

• Measurement memos on adoption/implementation/use, user effectiveness/productivity, and organizational performance/productivity due
• Begin working on draft measurement instrument

Designing an Evaluation Study

7, October 5: Introduction to research design; experimental and quasi-experimental designs; sampling; pilot testing of the instruments (notes, slides)

• Trochim, W. M. (2005). The Research Methods Knowledge Base. 2nd Edition. Internet WWW page, at URL: <http://trochim.human.cornell.edu/kb/index.htm> (version current as of January 16, 2005).
  • Probability sampling, http://www.socialresearchmethods.net/kb/sampprob.htm
  • Nonprobability sampling, http://www.socialresearchmethods.net/kb/sampnon.htm
• Gribbons, B., & Herman, J. (1997). True and quasi-experimental designs. Practical Assessment, Research & Evaluation, 5(14). http://PAREonline.net/getvn.asp?v=5&n=14.
• Dawson, T.E. (1997). A primer on experimental and quasi-experimental design. Paper presented at the annual meeting of the Southwest Educational Research Association, Austin, January, 1997. http://www.tele.sunyit.edu/expdes.HTM
• Optional: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research. Boston: Houghton Mifflin. [SILS - Q175 .C23]
• Designs 1-4 (pages 6-24), 6-7 (pages 25-26 and 37-43), and 10 (pages 47-50).
• Optional: Kerlinger, F. N. (2000). Foundations of Behavioral Research. Fourth edition. Fort Worth, TX: Harcourt College Publishers. [SILS - BF76.5 .K4 2000]
• Chapter 18, Inadequate designs and design criteria.

• Draft measurement instrument due
• Begin working on research design proposal

8, October 12: Interpretive approaches; Practical considerations in evaluation (notes, slide)

• Taylor, S. J., & Bogdan, R. (1984). Introduction to Qualitative Research Methods: The Search for Meanings. Second edition. New York: John Wiley & Sons. [SILS reserve - HM24 .T32 1984]
  • Chapter 1, Introduction: Go to the people, pages 1-12.
• Westbrook, L. (1994). Qualitative research methods: A review of major stages, data analysis techniques, and quality controls. Library & Information Science Research, 16(3), 241-254. [SILS reserve - PAM box]
• Mantei, M. M., & Teorey, T. J. (1989). Incorporating behavioral techniques into the systems development life cycle. MIS Quarterly, 13(3), 257-274. [Available via EBSCOhost]
• Work on research design proposal
• Specification of data needed for analysis

October 17, noon: Research design proposal due

• Data creation assignments will be distributed on October 18, to be completed by noon on October 23

Fall Break, October 17-20; no class Oct 19

Analyzing and Interpreting the Data

9, October 26: Introduction to data analysis; Preparing data for statistical analysis (notes)

• Trochim, W. M. (20050. The Research Methods Knowledge Base. 2nd Edition. Internet WWW page, at URL: <http://trochim.human.cornell.edu/kb/index.htm> (version current as of January 16, 2005).
  • Data preparation, http://www.socialresearchmethods.net/kb/statprep.htm
  • Conclusion validity, http://www.socialresearchmethods.net/kb/concval.htm
  • Threats to conclusion validity, http://www.socialresearchmethods.net/kb/concthre.htm
  • Improving conclusion validity, http://www.socialresearchmethods.net/kb/concimp.htm
  • Statistical power, http://www.socialresearchmethods.net/kb/power.htm

• Begin working on data analysis

10, November 2: Measures of central tendency and variability (meet in Computer Lab) (notes)

• Lane, D. M. (2005). HyperState Online Statistics Textbook. http://davidmlane.com/hyperstat/index.html
  • Chapter 2, Describing Univariate Data, http://davidmlane.com/hyperstat/desc_univ.html
    • Read pages on Mean, Median, Mode, Range, Semi-Interquartile Range, Variance, Standard Deviation
• Optional: Stempel, G. H., III, & Westley, B. H. (Eds.) (1989). Research Methods in Mass Communication. Englewood Cliffs, NJ: Prentice Hall. [SILS - P91 .R43 1989]
  • Basic Statistical Tools, by David H. Weaver, pages 49-89
• Optional: Morgan, S. E., Reichert, T., & Harrison, T. R. (2002). From Numbers to Words: Reporting Statistical Results for the Social Sciences. Boston: Allyn & Bacon. [SILS - HA29 .M83165 2002]
  • Chapter 3, Descriptive information, 23-26
• Optional: Salkind, N. J. (2000). Statistics for People Who (Think They) Hate Statistics. Thousand Oaks, CA: Sage Publications. [SILS - HA29 .S2365 2000]
  • Chapter 2, Means to an end: computing and understanding averages, 21-37
  • Chapter 3, Vive la différence: understanding variability, 38-52

11, November 9: Relationships between variables (meet in Computer Lab)

• Lane, D. M. (2005). HyperState Online Statistics Textbook. http://davidmlane.com/hyperstat/index.html
  • Chapter 3, Describing Bivariate Data, http://davidmlane.com/hyperstat/desc_biv.html
    • Read pages on Introduction to Pearson's Correlation, Spearman's rho
  • Chapter 10, Hypothesis Testing with Standard Errors, http://davidmlane.com/hyperstat/hypothesis_testing_se.html
    • Read pages on µ1 - µ2, independent groups, s estimated; µ1 - µ2, dependent means, s estimated
  • Chapter 16, Chi Square, http://davidmlane.com/hyperstat/chi_square.html
    • Read pages on Chi square test of independence: Introduction and Calculation
• Optional: Morgan, S. E., Reichert, T., & Harrison, T. R. (2002). From Numbers to Words: Reporting Statistical Results for the Social Sciences. Boston: Allyn & Bacon. [SILS - HA29 .M83165 2002]
  • Chapter 5, Correlation, 31-34
  • Chapter 6, Nonparametric statistics: Chi-square tests, 36-40
  • Chapter 7, Parametric statistics, t test and analysis of variance (ANOVA), 54-62
• Optional: Salkind, N. J. (2000). Statistics for People Who (Think They) Hate Statistics. Thousand Oaks, CA: Sage Publications. [SILS - HA29 .S2365 2000]
  • Chapter 5, Ice cream and crime: computing correlation coefficients, 85-108
  • Chapter 9, Significantly significant: what it means for you and me, 171-190
  • Chapter 10, t(ea) for two: tests between the means of different groups, 191-205
  • Chapter 11, t(ea) for two (again): tests between the means of related groups, 206-219
  • Chapter 12, Two groups too many? Try analysis of variance, 220-239
  • Chapter 13, Cousins or just good friends? Testing relationships using the correlation coefficient, 240-254
  • Chapter 14, What to do when you're not normal: chi-square and some other nonparametric tests, 255-267

12, November 16: Analysis of qualitative data (notes)

• Cavanagh, S. (1997). Content analysis: Concepts, methods and applications [Qualitative data analysis]. Nurse Researcher, 4(3), 5-16. [pdf distributed via email]

Drawing Conclusions from the Results

• Thompson, B. (1994). The concept of statistical significance testing. Practical Assessment, Research & Evaluation, 4(5). http://PAREonline.net/getvn.asp?v=4&n=5.
• Belcher, L. W., & Watson, H. J. (1993). Assessing the value of Conoco's EIS. MIS Quarterly, 17(3), 239-253. [Available via EBSCOhost]

• Data analysis due
• Begin working on final reports

• Oral presentation of management reports
• Written management reports due, Wednesday, December 13, before noon