Abstract
In our tutorial, we will share more than six years of our crowdsourced data labeling experience and bridge the gap between crowdsourcing and information retrieval communities by showing how one can incorporate human-in-the-loop into their retrieval system to gather the real human feedback on the model predictions. Most of the tutorial time is devoted to a hands-on practice, when the attendees will, under our guidance, implement an end-to-end process for information retrieval from problem statement and data labeling to machine learning model training and evaluation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chu, Z., Ma, J., Wang, H.: Learning from crowds by modeling common confusions. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35(7), 5832–5840 (2021). https://doi.org/10.1609/aaai.v35i7.16730
Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., Allahbakhsh, M.: Quality control in crowdsourcing: a survey of quality attributes, assessment techniques, and assurance actions. ACM Comput. Surv. 51(1), 7:1–7:40 (2018)
Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the EM algorithm. J. Roy. Stat. Soc. 28(1), 20–28 (1979). https://doi.org/10.2307/2346806
Rodrigues, F., Pereira, F.: Deep Learning from Crowds. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32(1) (2018). https://doi.org/10.1609/aaai.v32i1.11506
Ustalov, D., Pavlichenko, N., Losev, V., Giliazev, I., Tulin, E.: A general-purpose crowdsourcing computational quality control toolkit for python. In: The Ninth AAAI Conference on Human Computation and Crowdsourcing: Works-in-Progress and Demonstration Track, HCOMP 2021 (2021). https://arxiv.org/abs/2109.08584
Zheng, Y., Li, G., Li, Y., Shan, C., Cheng, R.: Truth inference in crowdsourcing: is the problem solved? Proceedings VLDB Endow. 10(5), 541–552 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ustalov, D., Smirnova, A., Fedorova, N., Pavlichenko, N. (2023). Crowdsourcing for Information Retrieval. In: Kamps, J., et al. Advances in Information Retrieval. ECIR 2023. Lecture Notes in Computer Science, vol 13982. Springer, Cham. https://doi.org/10.1007/978-3-031-28241-6_37
Download citation
DOI: https://doi.org/10.1007/978-3-031-28241-6_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-28240-9
Online ISBN: 978-3-031-28241-6
eBook Packages: Computer ScienceComputer Science (R0)