Evaluation Criteria of Information Retrieval Systems: What We Know and What We Do Not Know

Hariri, Nadjla; Babalhavaeji, Fahime; Farzandipour, Mehrdad; Nadi Ravandi, Somayyeh

doi:10.35050/JIPM010.2014.007

Evaluation Criteria of Information Retrieval Systems: What We Know and What We Do Not Know

Authors

Nadjla Hariri

Fahime Babalhavaeji

Mehrdad Farzandipour

Somayyeh Nadi Ravandi

10.35050/JIPM010.2014.007

Abstract

Evaluation of information retrieval systems is one of the
greatest challenges for information science specialists, because
determining the performance of a system depends on judgment of the
relevance of documents provided by the system to the user's
information needs, and it has its own complexities. New retrieval
systems due to the dynamic nature of the Web are very different
compared with traditional retrieval systems. Web information retrieval
systems in terms of ranking results are divided into two groups: The
ranked retrieval results and unranked retrieval sets. Each system has a
different evaluation criteria and scales.
In this paper, criteria for evaluating information retrieval systems is
reviewed for the ranked retrieval results (including precision and recall
curves, interpolated precision, the 11-point interpolated precision
average, Mean Average Precision, precision at K, R-precision, ROC
curve cumulative gain and normalized discounted cumulative gain), and
the unranked retrieval sets (including precision, recall, F-measure and
accuracy) is introduced separately. Finally, alluding to the evaluation
metrics, a scale of 4 degrees including best, useful, objective precision
and then differential precision is introduced which can be used to evaluate the non-binary precision of web information retrieval systems.

Keywords

information retrieval

precision

Evaluation

Evaluation Criterion

Ranked Retrieval Results

Unranked Retrieval Sets