Motivation
In recent years, immense progress has been made in the development of recommendation, retrieval and personalisation techniques. The evaluation of these systems is still based on traditional information retrieval and statistics metrics, e.g. precision, recall and/or RMSE, often not taking the use-case and situation of the system into consideration. However, the rapid evolution of recommender and adaptive IR systems in both their goals and their application domains foster the need for new evaluation methodologies and environments.
The workshop will be followed by a special issue of ACM TiST on Recommender System Benchmarking. Authors of high-quality papers from the workshop will be encouraged to submit extended versions of their work to the journal.
Submission topics
We invite the submission of papers reporting relevant research in the area of benchmarking and evaluation of recommendation and adaptive IR systems. We welcome submissions presenting contributions in this scope, addressing the following topics:
- New metrics and methods for quality estimation of recommender and adaptive IR systems
- Novel frameworks for the user-centric evaluation of adaptive systems
- Validation of off-line methods with online studies
- Comparison of evaluation metrics and methods
- Comparison of recommender and IR approaches across multiple systems and domains
- Measuring technical constraints vs. accuracy
- New datasets for the evaluation of recommender and adaptive IR systems
- Benchmarking frameworks
- Multiple-objective benchmarking
Submissions
We invite the submission of papers reporting original research, studies, advances, or experiences in this area. Two submission types are accepted: long papers of up to 8 pages, and short papers up to 4 pages, in the standard ACM SIG proceedings format. Paper submissions and reviews will be handled electronically.
Each paper will be evaluated by at least three reviewers from the Program Committee. The papers will be evaluated for their originality, contribution significance, soundness, clarity, and overall quality. The interest of contributions will be assessed in terms of technical and scientific findings, contribution to the knowledge and understanding of the problem, methodological advancements, or applicative value.
Submission instructions can be found on the Submissions page.
Important dates
- Paper submission: June 14, 2013
- Notification: June 28th, 2013
- Camera ready: July 13th, 2013
- Workshop: August 1st, 2013