Abstract
Negative results are not popular to disseminate. However, their publication would help to save resources and foster scientific communication. This study analysed the bibliometric and semantic nature of negative results publications. The Journal of Negative Results in Biomedicine (JNRBM) was used as a role model. Its complete articles from 2002–2009 were extracted from SCOPUS and supplemented by related records. Complementary negative results records were retrieved from Web of Science in “Biochemistry” and “Telecommunications”. Applied bibliometrics comprised of co-author and co-affiliation analysis and a citation impact profile. Bibliometrics showed that authorship is widely spread. A specific community for the publication of negative results in devoted literature is non-existent. Neither co-author nor co-affiliation analysis indicated strong interconnectivities. JNRBM articles are cited by a broad spectrum of journals rather than by specific titles. Devoted negative results journals like JNRBM have a rather low impact measured by the number of received citations. On the other hand, only one-third of the publications remain uncited, corroborating their importance for the scientific community. The semantic analysis relies on negative expressions manually identified in JNRBM article titles and abstracts and extracted to syntactic patterns. By using a Natural Language Processing tool these patterns are then employed to detect their occurrences in the multidisciplinary bibliographical database PASCAL. The translation of manually identified negation patterns to syntactic patterns and their application to multidisciplinary bibliographic databases (PASCAL, Web of Science) proved to be a successful method to retrieve even hidden negative results. There is proof that negative results are not only restricted to the biomedical domain. Interestingly a high percentage of the so far identified negative results papers were funded and therefore needed to be published. Thus policies that explicitly encourage or even mandate the publication of negative results could probably bring about a shift in the current scientific communication behaviour.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Available from: http://www.nooj4nlp.net. Accessed 23 April 2012.
In our study institution is defined as the top level entry of the database affiliation field. Manual disambiguation was done for the most relevant ones.
Available from: http://www.scimagojr.com. Accessed 23 April 2012.
e.g. The Journal of Biomedicine and Biotechnology has a comparable self-citation rate with 2 % (2010 JCR Science Edition).
References
Agarwal, S., & Yu, H. (2010). Biomedical negation scope detection with conditional random fields. Journal of the American Medical Informatics Association, 17(6), 696–701.
Agarwal, S., Yu, H., & Kohane, I. (2011). BioNOT: A searchable database of biomedical negated sentences. BMC Bioinformatics, 12, 420. doi:10.1186/1471-2105-12-420.
Aucouturier, J. J., & Pachet, F. (2004). Improving timbre similarity: How high is the sky? Journal of Negative Results in Speech and Audio Sciences, 1, 1.
Browman, H. I. (1999). Negative results. Marine Ecology Progress Series, 191, 301–309.
Chan, A. W., Hrobjartsson, A., Haahr, M. T., Gotzsche, P. C., & Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials—Comparison of protocols to published articles. JAMA, 291, 2457–2465.
De Vries, R., Anderson, M. S., & Martinson, B. C. (2006). Normal misbehavior: Scientists talk about the ethics of research. Journal of Empirical Research on Human Research Ethics, 1, 43–50.
Dickersin, K. (1990). The existence of publication bias and risk factors for its occurrence. JAMA, 263(10), 1385–1389.
Dickersin, K., Min, Y. I., & Meinert, C. L. (1992). Factors influencing publication of research results—follow-up of applications submitted to 2 institutional review boards. JAMA, 267, 374–378.
Easterbrook, P. J., Berlin, J. A., Gopalan, R., & Matthews, D. R. (1991). Publications bias in clinical research. Lancet, 337(8746), 867–872. doi:10.1016/0140-6736(91)90201-Y.
Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4, e5738.
Fanelli, D. (2010). Do pressures to publish increase scientists’ bias? An empirical support from US States data. PLoS ONE, 5(4), e10271.
Fanelli, D. (2011). Positive results increase down the hierarchy of the sciences. PLoS ONE,. doi:10.1371/journal.pone.0010068.
González-Pereira, B., Guerrero-Bote, V. P., Moya-Anegón, F. de (2010). A new approach to the metric of journals’ scientific prestige: The SJR indicator. Journal of Informetrics 4(3), 379–391. http://arxiv.org/ftp/arxiv/papers/0912/0912.4141.pdf. Accessed 23 April 2012.
Gupta, N., & Stopfer, M. (2011). Negative results need airing too. Nature, 470, 39. doi:10.1038/470039a.
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327, 560.
Jerrells, T. R. (2003). Publication of negative research findings: pros and cons. Alcohol, 31, 123–124.
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217.
Kopcsa, A., Schiebel, E. (2001). Content Visualisation by BibTechMon™. In Informatics 2001/Network Economy-Visions and Reality, OCG-Austrian Computer Society.
Lexchin, J., Bero, L. A., Djulbegovic, B., & Clark, O. (2003). Pharmaceutical industry sponsorship and research outcome and quality: Systematic review. British Medical Journal, 326, 1167.
Miller, S. C., & Moulder, J. E. (1998). Publication of negative results is an essential part of the scientific process. Radiation Research, 150(1), 3–16.
Moed, H. F. (2010). The source normalized impact per paper is a valid and sophisticated indicator of journal citation impact. Journal of the American Society for Information Science and Technology, 62(1), 211–213.
Moed, H. F. (2011). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265–277.
Morante, R., Daelemans, W. (2009). A metalearning approach to processing the scope of negation. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning, pages 21–29.
Newman, M. E. J. (2008). The mathematics of networks. The New Palgrave Encyclopedia of Economics. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.131.8175&rep=rep1&type=pdf (Accessed 2012-07-12).
Prechelt, L. (1997). Why we need an explicit forum for negative results. Journal of Universal Computer Science, 3, 1074–1083.
Scargle, J. D. (2000). Publication bias: the “file-drawer” problem in scientific inference. Journal of Scientific Exploration, 14(1), 91–106.
Smith, M. L. (1980). Publication bias and meta-analysis. Evaluation Educ, 4, 22–24.
Sterne, J. A. C., Egger, M., & Smith, G. D. (2001). Systematic reviews in health care—Investigating and dealing with publication and other biases in meta-analysis. British Medical Journal, 323, 101.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Gumpenberger, C., Gorraiz, J., Wieland, M. et al. Exploring the bibliometric and semantic nature of negative results. Scientometrics 95, 277–297 (2013). https://doi.org/10.1007/s11192-012-0829-z
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-012-0829-z