Hady W. Lauw
Ee-Peng Lim
ABSTRACT
In this paper, we investigate how deviation in evaluation activities may reveal bias on the part of reviewers and controversy on the part of evaluated objects. We focus on a 'data-centric approach' where the evaluation data is assumed to represent the 'ground truth'. The standard statistical approaches take evaluation and deviation at face value. We argue that attention should be paid to the subjectivity of evaluation, judging the evaluation score not just on 'what is being said' (deviation), but also on 'who says it' (reviewer) as well as on 'whom it is said about' (object). Furthermore, we observe that bias and controversy are mutually dependent, as there is more bias if there is higher deviation on a less controversial object. To address this mutual dependency, we propose a reinforcement model to identify bias and controversy. We test our model on real-life data to verify its applicability.
- H. Anton and C. Rorres. Elementary Linear Algebra with Applications. John Wiley & Sons, Inc., 1987.Google Scholar
- R. Fagin, R. Kumar, and D. Sivakumar. Comparing top k lists. SIAM Journal on Discrete Mathematics, 17(1):134--160, 2003. Google ScholarDigital Library
- C. Faloutsos, K. S. McCurley, and A. Tomkins. Fast discovery of connection subgraphs. In KDD, pages 118--127, 2004. Google ScholarDigital Library
- K. Faust. Centrality in affiliation networks. Social Networks, 19(2):157--191, 1997.Google ScholarCross Ref
- N. Franke, M. Gruber, D. Harhoff, and J. Henkel. What you are is what you like - similarity biases in venture capitalists' evaluations of start-up teams. Journal of Business Venturing, In Press, Corrected Proof, 2005.Google Scholar
- D. Gibson, J. Kleinberg, and P. Raghavan. Inferring Web communities from link topology. In Hypertext, pages 225--234, 1998. Google ScholarDigital Library
- G. H. Golub and C. F. Van Loan. Matrix Computations. Johns Hopkins University Press, 3rd edition, 1996.Google Scholar
- D. Kempe, J. Kleinberg, and E. Tardos. Maximizing the spread of influence through a social network. In KDD, pages 137--146, 2003. Google ScholarDigital Library
- J. M. Kleinberg. Authoritative sources in a hyperlinked environment. Journal of the ACM, 46(5):604--632, 1999. Google ScholarDigital Library
- E. Knorr and R. Ng. A unified notion of outliers: Properties and computation. In KDD, pages 219--222, 1997.Google Scholar
- Mathematica. Mathematica. http://www.wolfram.com/products/mathematica/index.html.Google Scholar
- A. McCook. Is peer review broken? The Scientist, 20(2):26, 2006.Google Scholar
- F. Moers. Discretion and bias in performance evaluation: The impact of diversity and subjectivity. Accounting, Organizations and Society, 30(1):67--80, 2005.Google ScholarCross Ref
- L. Page, S. Brin, R. Motwani, and T. Winograd. The PageRank citation ranking: Bringing order to the Web. In Stanford Digital Library Technologies Project, 1998.Google Scholar
- G. Ruskin. Commercial Alert asks FTC to investigate buzz marketers for deception. Retrieved February 17, 2006, from http://www.commercialalert.org/news/news-releases/2005/10/commercial-alert-asks-ftc-to-investigate-buzz-marketers-for-deception, 2005.Google Scholar
- J. Sun, H. Qu, D. Chakrabarti, and C. Faloutsos. Neighborhood formation and anomaly detection in bipartite graphs. In ICDM, pages 418--425, 2005. Google ScholarDigital Library
- K. Wang and M.-Y. T. Su. Item selection by "hub-authority" profit ranking. In KDD, pages 652--657, 2002. Google ScholarDigital Library
Index Terms
- Bias and controversy: beyond the statistical deviation
Recommendations
Modeling Controversy within Populations
ICTIR '17: Proceedings of the ACM SIGIR International Conference on Theory of Information RetrievalA growing body of research focuses on computationally detecting controversial topics and understanding the stances people hold on them. Yet gaps remain in our theoretical and practical understanding of how to define controversy, how it manifests, and ...
Bias in peer review
Research on bias in peer review examines scholarly communication and funding processes to assess the epistemic and social legitimacy of the mechanisms by which knowledge communities vet and self-regulate their work. Despite vocal concerns, a closer look ...
Bias and Controversy in Evaluation Systems
Evaluation is prevalent in real-life. With the advent of Web 2.0, online evaluation has become an important feature in many applications that involve information (e.g., video, photo, audio) sharing, and social networking (e.g., blogging). In these ...
Comments