Enrique Amigó
ABSTRACT
The EvALL online evaluation service aims to provide a unified evaluation framework for Information Access systems that makes results completely comparable and publicly available for the whole research community. For researchers working on a given test collection, the framework allows to: (i) evaluate results in a way compliant with measurement theory and with state-of-the-art evaluation practices in the field; (ii) quantitatively and qualitatively compare their results with the state of the art; (iii) provide their results as reusable data to the scientific community; (iv) automatically generate evaluation figures and (low-level) interpretation of the results, both as a pdf report and as a latex source. For researchers running a challenge (a comparative evaluation campaign on shared data), the framework helps them to manage, store and evaluate submissions, and to preserve ground truth and system output data for future use by the research community. EvALL can be tested at http://evall.uned.es.
- Robert N. Allan. 2009. Virtual research environments: From portals to science gateways. Elsevier. Google Scholar
Cross Ref
- Enrique Amigó, Julio Gonzalo, Javier Artiles, and Felisa Verdejo 2009. A comparison of extrinsic clustering evaluation metrics based on formal constraints. Information retrieval Vol. 12, 4 (2009), 461--486. Google ScholarDigital Library
- Enrique Amigó, Julio Gonzalo, and Felisa Verdejo. 2013. A general evaluation measure for document organization tasks Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. ACM, 643--652.Google Scholar
- Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel 2009. Improvements That Don't Add Up: Ad-hoc Retrieval Results Since 1998 Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM '09). ACM, New York, NY, USA, 601--610. http://dx.doi.org/10.1145/1645953.1646031 Google ScholarDigital Library
- C. Buckley and others. 2004. The trec_eval evaluation package. (2004).Google Scholar
- Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: an update. ACM SIGKDD explorations newsletter Vol. 11, 1 (2009), 10--18.Google ScholarDigital Library
- Jimmy Lin and Miles Efron 2014. Infrastructure Support for Evaluation As a Service Proceedings of the 23rd International Conference on World Wide Web (WWW '14 Companion). ACM, New York, NY, USA, 79--82. http://dx.doi.org/10.1145/2567948.2577014 Google ScholarDigital Library
- R. Usbeck, M. Roder, and A. Ngonga 2015. GERBIL - General Entity Annotator Benchmarking Framework Proc. 25th World Wide Web Conference. ACM.Google Scholar
- WFMC 1994. Workflow reference model. Technical Report. Workflow Management Coalition, Brussels.Google Scholar
- Peilin Yang and Hui Fang 2016. A Reproducibility Study of Information Retrieval Models Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval (ICTIR '16). ACM, New York, NY, USA, 77--86. http://dx.doi.org/10.1145/2970398.2970415 Google ScholarDigital Library
Recommendations
Click the search button and be happy: evaluating direct and immediate information access
CIKM '11: Proceedings of the 20th ACM international conference on Information and knowledge managementWe define Direct Information Access as a type of information access where there is no user operation such as clicking or scrolling between the user's click on the search button and the user's information acquisition; we define Immediate Information ...
A new metric to compare local community detection algorithms in social networks using geodesic distance
AbstractCommunity detection problem is a well-studied problem in social networks. One major question to this problem is how to evaluate different community detection algorithms. This issue is even more challenging in the problem of local community ...
On the choice of effectiveness measures for learning to rank
AbstractMost current machine learning methods for building search engines are based on the assumption that there is a target evaluation metric that evaluates the quality of the search engine with respect to an end user and the engine should be trained to ...
Comments