Skip to main content
SpringerLink
Log in

Reproducible research in computational economics: guidelines, integrated approaches, and open source software

  • Published:
Computational Economics Aims and scope Submit manuscript

Abstract

Traditionally, computer and software applications have been used by economists to off-load otherwise complex or tedious tasks onto technology, freeing up time and intellect to address other, intellectually more rewarding, aspects of research. On the negative side, this increasing dependence on computers has resulted in research that has become increasingly difficult to replicate. In this paper, we propose some basic standards to improve the production and reporting of computational results in economics for the purpose of accuracy and reproducibility. In particular, we make recommendations on four aspects of the process: computational practice, published reporting, supporting documentation, and visualization. Also, we reflect on current developments in the practice of computing and visualization, such as integrated dynamic electronic documents, distributed computing systems, open source software, and their potential usefulness in making computational and empirical research in economics more easily reproducible.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Amman H., Kendrick D., Rust J. (eds). (1996). Handbook of computational economics (Vol 1). Amsterdam The Netherlands, Elsevier North-Holland

    Google Scholar 

  • Anderson R.G., Greene W.H., McCullough B., & Vinod H.D. (2005). The role of data and program code archives in the future of economic research. Working Paper No. 2005-014B, FRB of St. Louis.

  • Baiocchi G. (2003). Managing econometric projects using Perl. Journal of Applied Econometrics 18(3): 371–378

    Article  Google Scholar 

  • Baiocchi G. (2004). Using Perl for statistics: Data processing and statistical computing. Journal of Statistical Software 11(1): 1–81

    Google Scholar 

  • Baiocchi G., Distaso W. (2003). GRETL: Econometric software for the GNU generation. Journal of Applied Econometrics 18(1): 105–110

    Article  Google Scholar 

  • Belsey D.A., Kuh E., Welsch R.E. (1980). Regression diagnostics. New York, Wiley

    Google Scholar 

  • Buckheit J.B., Donoho D.L. (1995). Wavelets and statistics, chapter Wavelab and Reproducible Research. Berlin, New York, Springer, pp. 55–81

    Google Scholar 

  • Cleveland W.S. (1994). The elements of graphing data. Summit, New Jersey, Hobart Press

    Google Scholar 

  • Cleveland W.S. (1993). Visualizing data. Summit, New Jersey, Hobart Press

    Google Scholar 

  • de Leeuw J. (2005). On abandoning XLISP-STAT. Journal of Statistical Software 13(7): 1–81

    Google Scholar 

  • Dewald W.G., Thursby J.G., Anderson R.G. (1986). Replication in empirical economics: The journal of money, credit and banking project. American Economic Review 76(4): 587–603

    Google Scholar 

  • Eddelbuettel D. (2000). Econometrics with Octave. Journal of Applied Econometrics 15(5): 531–542

    Article  Google Scholar 

  • Eddelbuettel D. (2003). Quantian: A scientific computing environment. In Proceedings of the 3rd international workshop on distributed statistical computing (DSC 2003) March 20–22, Vienna, Austria, Vienna, Austria. Technische Universitt Wien.

  • Gentle J.E. (2003). Random number generation and Monte Carlo methods (2nd ed). New York, Springer

    Google Scholar 

  • Gentleman R. (2004). Some perspectives on statistical computing. Technical report, Department of Biostatistics, Harvard School of Public Health, Boston, Massachusetts, USA.

  • Gentleman R. (2005). Reproducible research: A bioinformatics case study. Statistical Applications in Genetics and Molecular Biology, 4(1), Article 2. Available at: http://www.bepress.com/sagmb/vol4/iss1/art2.

  • Gentleman R., & Lang D.T. (2004).Statistical analyses and reproducible research. Bioconductor Project Working Papers. Working Paper 2

  • Greene W. (2000). Econometric analysis (4th ed). New York, Prentice Hall

    Google Scholar 

  • Hoaglin D.C., Andrews D.F. (1975). The reporting of computation-based results in statistics. The American Statistician 29(3): 122–126

    Article  Google Scholar 

  • Ihaka R., Gentleman R. (1996). R: A language for data analysis and graphics. Journal of Computational and Graphical Statistics 5, 299–314

    Article  Google Scholar 

  • Judd K., Tesfatsion L. (eds). (2006). Handbook of computational economics: Agent-based computational economics (Vol 2). Amsterdam, The Netherlands, Elsevier North-Holland

    Google Scholar 

  • Kernighan B.W., Pike R. (1999). The practice of programming. Reading, MA, Addison-Wesley

    Google Scholar 

  • Kernighan B.W., Plauger P.J. (1978). The elements of programming style (2nd ed). New York, NY, McGraw Hill

    Google Scholar 

  • Knopper K. (2003). Knoppix. Available at http://www.knopper.net/knoppix/index-en.html.

  • Knüsel L. (1995). On the accuracy of statistical distributions in GAUSS. Computational Statistics and Data Analysis 20, 699–702

    Article  Google Scholar 

  • Knuth D.E. (1983). Literate programming. Technical report STAN-CS-83-981, Stanford University, Department of Computer Science.

  • Knuth D.E. (1984). Literate programming. The Computer Journal 27(2): 97–111

    Article  Google Scholar 

  • Knuth D.E. (1992). Literate programming. CSLI Lecture Notes Number 27 Stanford, CA, USA: Stanford University Center for the Study of Language and Information.

  • Koenker R. (1996). Reproducible econometric research. Technical report, Department of Econometrics, University of Illinois, Urbana-Champaign, IL.

  • Koenker R. (2006). Reproducibility in econometrics research. Technical report, Department of Econometrics, University of Illinois, Urbana-Champaign, IL. http://www.econ.uiuc.edu/∼ roger/repro.html.

  • Leisch F. (2002). Dynamic generation of statistical reports using literate data analysis. In: Härdle W. (eds), Proceedings in computational statistics. Heidelberg Germany, Physika Verlag, pp. 575–580

    Google Scholar 

  • Leontief W.W. (1966). Input-output economics. In: Leontief W.W. (eds), Input-output economics chapter 2. New York, Oxford University Press, pp. 13–29

    Google Scholar 

  • Lerner J., Triole J. (2002). The simple economics of open source. Journal of Industrial Economics 52, 197–234

    Google Scholar 

  • MacKinnon J.G. (1999). The Linux operating system: Debian GNU/Linux. Journal of Applied Econometrics 14(4): 443–452

    Article  Google Scholar 

  • McCullough B. (1998). Assessing the reliability of statistical software: Part I. The American Statistician 52, 358–366

    Article  Google Scholar 

  • McCullough B. (1999). Assessing the reliability of statistical software: Part II. The American Statistician 53(1): 149–159

    Article  Google Scholar 

  • McCullough B., Vinod H. (1999a). The numerical reliability of econometric software. Journal of Economic Literature 37(2): 633–665

    Google Scholar 

  • McCullough B., Vinod H. (1999b). The numerical reliability of econometric software. Journal of Economic Literature XXXVII: 633–665

    Google Scholar 

  • McCullough B.D., McGeary K.A., Harrison T.D. (2006). Lessons from the JMCB Archive. Journal of Money, Credit, and Banking 38(4): 1093–1107

    Article  Google Scholar 

  • McCullough B.D., Renfro C.G. (1999). Benchmarks and software standards: A case study of GARCH procedures. Journal of Economic and Social Measurement 25(2): 59–71

    Google Scholar 

  • Racine J. (2000). The cygwin tools: a GNU toolkit for windows. Journal of Applied Econometrics 15(3): 331–341

    Article  Google Scholar 

  • Racine J., Hyndman R. (2002). Using R to teach econometrics. Journal of Applied Econometrics 17(2): 175–189

    Article  Google Scholar 

  • Ramanathan R. (2002). Introductory econometrics with applications (5th ed). Orlando Florida, Harcourt College Publishers

    Google Scholar 

  • Rossini A.J., Heiberger R.M., Sparapani R., Mächler M., Hornik K.(2004). Emacs speaks statistics: A multiplatform, multi-package development environment for statistical analysis. Journal of Computational and Graphical Statistics 13(1): 247–261

    Article  Google Scholar 

  • Sawitzki G. (1999). Software components and document integration for statistical computing. In Proceedings ISI Helsinki 1999 (52nd session) Bulletin of the International Statistical Institute Tome LVIII, Book 2, pp. 117–120

  • Sawitzki G. (2005). Keeping statistics alive in documents. Computational Statistics 17(1): 65–88

    Article  Google Scholar 

  • Schwab M., Karrenbach M., Claerbout J. (2000). Making scientific computations reproducible. Computing in Science and Engineering 2(6): 61–67

    Article  Google Scholar 

  • St. Laurent A.M. (2004). Understanding open source and free software licensing: A straightforward guide to the complex world of licensing. Sebastopol, CA, USA: O’Reilly & Associates.

  • Stallman R. (1985). The GNU manifesto. Dr. Dobb’s Journal of Software Tools 10(3): 30–35

    Google Scholar 

  • Stokes H. (2004). On the advantage of using two or more econometric software systems to solve the same problem. Journal of Economic and Social Measurement 29, 307–320

    Google Scholar 

  • Tufte E.R. (2001). The visual display of quantitative information (2nd ed). Cheshire Connecticut, Graphics Press

    Google Scholar 

  • Ueberhuber C.W. (1997). Numerical computation: Methods, software, and analysis (Vol 1). Berlin Heidelberg, Germany, Springer

    Google Scholar 

  • Varian H.R. (eds). (1992). Economic and financial modeling with mathematica. New York, TELOS/Springer

    Google Scholar 

  • Varian H.R. (eds). (1996). Computational economics: Economic and financial analysis with mathematica. New York, TELOS/Springer

    Google Scholar 

  • Vinod H.D. (2000). Review of GAUSS for windows, including its numerical accuracy. Journal of Applied Econometrics 14(2): 211–220

    Article  Google Scholar 

  • Vinod H.D. (2001). Care and feeding of reproducible econometrics. Journal of Econometrics 100(1): 87–88

    Article  Google Scholar 

  • Wooldridge J. (2002). Introductory econometrics: A modern approach (2nd ed). Mason OH, Thomson South-Western

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Economics and Finance, University of Durham, Durham, DH1 3HY, UK

    Giovanni Baiocchi

Authors
  1. Giovanni Baiocchi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giovanni Baiocchi.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Baiocchi, G. Reproducible research in computational economics: guidelines, integrated approaches, and open source software. Comput Econ 30, 19–40 (2007). https://doi.org/10.1007/s10614-007-9084-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10614-007-9084-4

Keywords

JEL classification

Search

Navigation