Evaluating collaborative filtering recommender systems

Authors : Jonathan L. Herlocker , Joseph A. Konstan , Loren G. Terveen , John T. Riedl Authors Info & Claims

New Citation Alert added!

This alert has been successfully added and will be sent to: You will be notified whenever a record that you have chosen has been cited.

New Citation Alert!

• Information & Contributors
• Bibliometrics & Citations
• Get Access
• References
• Media
• Tables
• Share

Abstract

Recommender systems have been evaluated in many, often incomparable, ways. In this article, we review the key decisions in evaluating collaborative filtering recommender systems: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes other than quality, and the user-based evaluation of the system as a whole. In addition to reviewing the evaluation strategies used by prior researchers, we present empirical results from the analysis of various accuracy metrics on one content domain where all the tested metrics collapsed roughly into three equivalence classes. Metrics within each equivalency class were strongly correlated, while metrics from different equivalency classes were uncorrelated.

References

Aggarwal, C. C., Wolf, J. L., Wu, K.-L., and Yu, P. S. 1999. Horting hatches an egg: A new graph-theoretic approach to collaborative filtering. In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, New York.]]

Amento, B., Terveen, L., Hill, W., Hix, D., and Schulman, R. 2003. Experiments in social data mining: The TopicShop System. ACM Trans. Computer-Human Interact. 10, 1 (Mar.), 54--85.]]

Amento, B., Terveen, L., Hix, D., and Ju, P. 1999. An empirical evaluation of user interfaces for topic management of web sites. In Proceedings of the 1999 Conference on Human Factors in Computing Systems (CHI '99). ACM, New York, 552--559.]]

Baeza-Yates, R. and Ribiero-Neto, B. 1999. Modern Information Retrieval. Addison-Wesley Longman, Boston, Mass.]]

Bailey, B. P., Gurak, L. J., and Konstan, J. A. 2001. An examination of trust production in computer-mediated exchange. In Proceedings of the 7th Conference on Human Factors and the Web (July).]]

BalabanovíC, M. and Shoham, Y. 1997. Fab: Content-based, collaborative recommendation. Commun. ACM 40, 66--72.]]

Basu, C., Hirsh, H., Cohen, W. W. 1998. Recommendation as classification: using social and content-based information in recommendation. In Proceedings of the 15th National Conference on Artificial Intelligence (AAAI-98). C. Rich, and J. Mostow, Eds. AAAI Press, Menlo Park, Calif., 714--720.]]

Billsus, D. and Pazzani, M. J. 1998. Learning collaborative information filters. In Proceedings of the 15th National Conference on Artificial Intelligence (AAAI-98). C. Rich, and J. Mostow, Eds. AAAI Press, Menlo Park, Calif., 46--53.]]

Breese, J. S., Heckerman, D., and Kadie, C. 1998. Empirical analysis of predictive algorithms for collaborative filtering. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence (UAI-98). G. F. Cooper, and S. Moral, Eds. Morgan-Kaufmann, San Francisco, Calif., 43--52.]]

Canny, J. 2002. Collaborative filtering with privacy via factor analysis. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information retrieval. ACM, New York, 238--245.]]

Claypool, M., Brown, D., Le, P., and Waseda, M. 2001. Inferring user Interest. IEEE Internet Comput. 5, 32--39.]]

Cleverdon, C. and Kean, M. 1968. Factors Determining the Performance of Indexing Systems. Aslib Cranfield Research Project, Cranfield, England.]]

Cosley, D., Lam, S. K., Albert, I., Konstan, J. A., and Riedl, J. 2003. Is seeing believing? How recommender interfaces affect users' opinions. CHI Lett. 5.]]

Dahlen, B. J., Konstan, J. A., Herlocker, J. L., Good, N., Borchers, A., and Riedl, J. 1998. Jump-starting movielens: User benefits of starting a collaborative filtering system with "dead data". TR 98-017. University of Minnesota.]]

Domingos, P. and Richardson, M. 2003. Mining the network value of customers. In Proceedings of the 7th International Conference on Knowledge Discovery and Data Mining. ACM, New York, 57--66.]]

Goldberg, D., Nichols, D., Oki, B. M., and Terry, D. 1992. Using collaborative filtering to weave an information tapestry. Commun. ACM 35, 61--70.]]

Goldberg, K., Roeder, T., Guptra, D., and Perkins, C. 2001. Eigentaste: A constant-time collaborative filtering algorithm. Inf. Retr. 4, 133--151.]]

Good, N., Schafer, J. B., Konstan, J. A., Borchers, A., Sarwar, B. M., Herlocker, J. L., and Riedl, J. 1999. Combining collaborative filtering with personal agents for better recommendations. In Proceedings of the 16th National Conference on Artificial Intelligence (AAAI-99), J. Hendler, and D. Subramanian, Eds. AAAI Press, Menlo Park, Calif., 439--446.]]

Hanley, J. A. and Mcneil, B. J. 1982. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29--36.]]

Harman, D. 1995. The TREC conferences. Hypertext---Information Retrieval---Multimedia: Synergieeffekte Elektronisher Informationssysteme. In Proceedings of HIM '95.]]

Harter, S. P. 1996. Variations in relevance assessments and the measurement of retrieval effectiveness. J. ASIS 47, 37--49.]]

Heckerman, D., Chickering, D. M., Meek, C., Rounthwaite, R., and Kadie, C. 2000. Dependency networks for inference, collaborative filtering, and data visualization. J. Mach. Learn. Res. 1, 49--75.]]

Herlocker, J. L., Konstan, J. A., Borchers, A., and Riedl, J. 1999. An algorithmic framework for performing collaborative filtering. In Proceedings of the 22nd International Conference on Research and Development in Information Retrieval (SIGIR '99) (Aug). M. A. Hearst, F. F. Gey, and R. Tong, Eds. ACM, New York. 230--237.]]

Herlocker, J. L., Konstan, J. A., and Riedl, J. 2000. Explaining collaborative filtering recommendations. In Proceedings of the 2000 Conference on Computer Supported Cooperative Work, 241--250.]]

Herlocker, J. L., Konstan, J. A., and Riedl, J. 2002. An empirical analysis of design choices in neighborhood-based collaborative filtering algorithms. Inf. Retr. 5, 287--310.]]

Hill, W., Stead, L., Rosenstein, M., and Furnas, G. W. 1995. Recommending and evaluating choices in a virtual community of use. In Proceedings of ACM CHI'95 Conference on Human Factors in Computing Systems. ACM, New York, 194--201.]]

Konstan, J. A., Miller, B. N., Maltz, D., Herlocker, J. L., Gordon, L. R., and Riedl, J. 1997. GroupLens: Applying collaborative filtering to usenet news. Commun. ACM 40, 77--87.]]

Le, C. T., Lindren, B. R. 1995. Construction and comparison of two receiver operating characteristics curves derived from the same samples. Biom. J. 37, 869--877.]]

Linton, F., Charron, A., and Joy, D. 1998. OWL: A recommender system for organziation-wide learning. In Proceedings of the 1998 Workshop on Recommender Systems 65--69.]]

McDonald, D. W. 2001. Evaluating Expertise Recommendations. In Proceedings of the ACM 2001 International Conference on Supporting Group Work (GROUP'01). ACM, New York.]]

Mcnee, S., Albert, I., Cosley, D., Gopalkrishnan, P., Rashid, A. M., Konstan, J. A., and Riedl, J. 2002. On the recommending of citations for research papers. In Proceedings of ACM CSCW 2002. ACM, New York.]]

Miller, B. N., Albert, I., Lam, S. K., Konstan, J. A., and Riedl, J. 2003. MovieLens unplugged: Experiences with a recommender systems on four mobile devices. In Proceedings of the 2003 Conference on Intelligent User Interfaces.]]

Miller, B. N., Riedl, J., and Konstan, J. A. 1997. Experiences with GroupLens: Making Usenet useful again. In Proceedings of the 1997 Usenix Technical Conference.]]

Mobasher, B., Dai, H., Luo, T., and Nakagawa, M. 2001. Effective personalization based on association rule discovery from web usage data. In Proceedings of the 3rd ACM Workshop on Web Information and Data Management (WIDM01), held in conjunction with the International Conference on Information and Knowledge Management (CIKM 2001). ACM, New York.]]

Morita, M. and Shinoda, Y. 1994. Information filtering based on user behavior analysis and best match text retrieval. In Proceedings of SIGIR '94, ACM, New York. 272--281.]]

Mui, L., Ang, C., and Mohtashemi, M. 2001. A Probabilistic Model for Collaborative Sanctioning. Technical Memorandum 617. MIT LCS.]]

Newman, W. 1997. Better or just different? On the benefits of designing interactive systems in terms of critical parameters. In Proceedings of the Designing Interactive Systems (DIS97). ACM, New York, 239--246.]]

Pennock, D. M., Horvitz, E., Lawrence, S., and Giles, C. L. 2000. Collaborative filtering by personality diagnosis: A hybrid memory- and model-based approach. In Proceedings of the 16th Annual Conference on Uncertainty in Artificial Intelligence (UAI-2000). Morgan Kaufmann, San Francisco, Calif., 473--480.]]

Rashid, A. M., Albert, I., Cosley, D., Lam, S. K., Mcnee, S., Konstan, J. A., and Riedl, J. 2002. Getting to know you: Learning new user preferences in recommender systems. In Proceedings of the 2002 Conference on Intelligent User Interfaces (IUI 2002). 127--134.]]

Reddy, P. K., Kitsuregawa, P., Sreekanth, P., and Rao, S. S. 2002. A graph based approach to extract a neighborhood customer community for collaborative filtering. In Databases in Networked Information Systems, Second International Workshop. Lecture Notes in Computer Science Springer-Verlag, New York, 188--200.]]

Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., and Riedl, J. 1994. GroupLens: An open architecture for collaborative filtering of netnews. In Proceedings of the 1994 Conference on Computer Supported Collaborative Work. R. Furuta and C. Neuwirth, Eds. ACM, New York. 175--186.]]