Skip to main content
Log in

The role of scaling up research in designing for and evaluating robustness

  • Published:
Educational Studies in Mathematics Aims and scope Submit manuscript

Abstract

One of the great strengths of Jim Kaput’s research program was his relentless drive towards scaling up his innovative approach to teaching the mathematics of change and variation. The SimCalc mission, “democratizing access to the mathematics of change,” was enacted by deliberate efforts to reach an increasing number of teachers and students each year. Further, Kaput asked: What can we learn from research at the next level of scale (e.g., beyond a few classrooms at a time) that we cannot learn from other sources? In this article, we develop an argument that scaling up research can contribute important new knowledge by focusing researchers’ attention on the robustness of an innovation when used by varied students, teachers, classrooms, schools, and regions. The concept of robustness requires additional discipline both in the design process and in the conduct of valid research. By examining a progression of three studies in the Scaling Up SimCalc program, we articulate how scaling up research can contribute to designing for and evaluating robustness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. Educational Service Centers are public regional organizations that offer educational support programs, ranging from financial or personnel support to innovative professional and curriculum, to districts throughout the state of Texas.

References

  • Assude, T., & Gelis, J. M. (2003). La dialectique ancien-nouveau dans l’intégration de Cabri-géomètre à l’école primaire. Educational Studies in Mathematics, 50(3), 259–287.

    Article  Google Scholar 

  • Baker, E. L. (2007). Principles for scaling up: Choosing, measuring effects, and promoting the widespread use of educational innovation. In B. Schneider & S.-K. McDonald (Eds.), Scale-up in education (pp. 37–54). Lanham, MD: Rowman & Littlefield.

    Google Scholar 

  • Ball, D. L., Hill, H. C., & Bass, H. (2005). Knowing mathematics for teaching: Who knows mathematics well enough to teach third grade, and how can we decide. American Educator, 29(3), 14–17, 20–22, 43–46.

    Google Scholar 

  • Brown, A. (1991). Design experiments. Theoretical and methodological challenges in evaluating complex interventions in classroom settings. Journal of the Learning Sciences, 2(2), 141–178.

    Article  Google Scholar 

  • Cerulli, M., Georget, J. P., Maracci, M., Psycharis, G., & Trgalova, J. (2007). Integrating research teams: The TEMLA approach. Retrieved on September 12, 2007, from http://telearn.noe-kaleidoscope.org/warehouse/TELMA_CERME5_conference_version.pdf.

  • Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9–13.

    Article  Google Scholar 

  • Coburn, C. E. (2003). Rethinking scale: Moving beyond numbers to deep and lasting change. Educational Researcher, 32(6), 3–12.

    Article  Google Scholar 

  • Cohen, D. K., Raudenbush, S., & Ball, D. L. (2003). Resources, instruction, and research. Educational Evaluation and Policy Analysis, 25(2), 1–24.

    Article  Google Scholar 

  • Cook, T. D. (1999). Considering the major arguments against random assignment: An analysis of the intellectual culture surrounding evaluation in American schools of education. Evanston, IL: Institute for Policy Research at Northwestern University.

    Google Scholar 

  • Dede, C. (2006). Scaling up: Evolving innovations beyond ideal settings to challenging contexts of practice. In R. K. Sawyer (Ed.), Cambridge handbook of learning sciences (pp. 551–566). Cambridge, UK: Cambridge University Press.

    Google Scholar 

  • Elmore, R. F. (1996). Getting to scale with good educational practice. Harvard Educational Review, 66(1), 1–26.

    Google Scholar 

  • Fishman, B. J., Marx, R. W., Best, S., & Tal, R. T. (2003). Linking teacher and student learning to improve professional development in systemic reform. Teaching and Teacher Education, 19, 643–658.

    Article  Google Scholar 

  • Fullan, M., & Earl, L. (2002). Large scale reform. Journal of Educational Change, 3, 1–5.

    Article  Google Scholar 

  • Hawkins, J. (1997). The national design experiments consortium:Final report. New York: Center for Children and Technology, Educational Development Center.

    Google Scholar 

  • Hedges, L. V. (2007). Generalizability of treatment effects: Psychometrics and education. In B. Schneider & S.-K. McDonald (Eds.), Scale-up in education (pp. 55–78). Lanham, MD: Rowman & Littlefield.

    Google Scholar 

  • Hiebert, J., Gallimore, R., Garnier, H., Givvin, K. B., Hollingsworth, H., Jacobs, J., et al. (2003). Teaching mathematics in seven countries: Results from the TIMSS 1999 video study. Washington DC: National Center for Educational Statistics.

    Google Scholar 

  • Kaput, J. (1992). Technology and mathematics education. In D. Grouws (Ed.), A handbook of research on mathematics teaching and learning (pp. 515–556). New York: Macmillan.

    Google Scholar 

  • Kaput, J. (1994). Democratizing access to calculus: New routes using old roots. In A. Schoenfeld (Ed.), Mathematical thinking and problem solving (pp. 77–155). Hillsdale, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Kaput, J. (1997). Rethinking calculus: Learning and thinking. The American Mathematical Monthly, 104(8), 731–737.

    Article  Google Scholar 

  • Kaput, J. (2000). Implications of the shift from isolated, expensive technology to connected, inexpensive, diverse and ubiquitous technologies. In M. O. J. Thomas (Ed.), Proceedings of the TIME 2000: An International Conference on Technology in Mathematics Education (pp. 1–24). Auckland, New Zealand: The University of Auckland and the Auckland University of Technology Also published in the New Zealand Mathematics Magazine, 38(3), December 2001.

    Google Scholar 

  • Kaput, J., & Roschelle, J. (1998). The mathematics of change and variation from a millennial perspective: New content, new context. In C. Hoyles, C. Morgan, & G. Woodhouse (Eds.), Rethinking the mathematics curriculum (pp. 155–170). London, UK: Falmer.

    Google Scholar 

  • Kaput, J., & Roschelle, J. (2000). Shifting representational infrastructures and reconstituting content to democratize access to the math of change and variation: Impacts on cognition, curriculum, learning and teaching. Paper presented at the NSF Workshop to Integrate Computer-based Modeling and Scientific Visualization into K-12 Teacher Education Programs. Reston, VA: National Science Foundation.

    Google Scholar 

  • Kaput, J., & Shaffer, D. (2002). On the development of human representational competence from an evolutionary point of view: From episodic to virtual culture. In K. Gravemeijer, R. Lehrer, B. van Oers, & L. Verschaffel (Eds.), Symbolizing, modeling and tool use in mathematics education (pp. 277–293). London: Kluwer Academic.

    Google Scholar 

  • Lagrange, J. B., Artigue, M., Laborde, C., & Trouche, L. (2003). Technology and mathematics education: A multidimensional study of the evolution of research and innovation. In A. J. Bishop, M.A. Clements, C. Keitel, J. Kilpatrick, & F. S. Leung (Eds.), Second international handbook of research in mathematics education (pp. 239–271). Dordrecht, The Netherlands: Kluwer Academic.

    Google Scholar 

  • National Research Council (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academies Press.

    Google Scholar 

  • O’Neil, J. (1995). Teacher and technology: Potential pitfalls. Educational Leadership, 53(2), 10–11.

    Google Scholar 

  • Porter, A. C. (2002). Measuring the content of instruction: Uses in research and practice. Educational Researcher, 31(7), 3–14.

    Article  Google Scholar 

  • Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.

    Google Scholar 

  • Reichardt, C. S. (2007). Estimating the effects of educational interventions. In B. Schneider & S.-K. McDonald (Eds.), Scale-up in education (Vol. 1, pp. 79–99). Lanham, MD: Rowman & Littlefield.

    Google Scholar 

  • Rogers, E. M. (2003). Diffusion of innovations. New York: Simon and Schuster.

    Google Scholar 

  • Romberg, T. A., & Kaput, J. (1999). Mathematics worth teaching, mathematics worth understanding. In E. Fennema & T. A. Romberg (Eds.), Mathematics classrooms that promote understanding (pp. 3–17). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Roschelle, J., & Jackiw, N. (2000). Technology design as educational research: Interweaving imagination, inquiry & impact. In A. Kelly & R. Lesh (Eds.), Handbook of research design in mathematics and science education (pp. 777–797). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Roschelle, J., & Kaput, J. (1996). Educational software architecture and systemic impact: The promise of component software. Journal of Educational Computing Research, 14(3), 217–228.

    Article  Google Scholar 

  • Roschelle, J., Kaput, J., & Stroup, W. (2000). SimCalc: Accelerating student engagement with the mathematics of change. In M. J. Jacobsen & R. B. Kozma (Eds.), Learning the sciences of the 21st century:Research, design, and implementing advanced technology learning environments (pp. 47–75). Hillsdale, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Roschelle, J., Kaput, J., Stroup, W., & Kahn, T. (1998). Scaleable integration of educational software: Exploring the promise of component architectures. Retrieved January 8, 2003, from http://www-jime.open.ac.uk/98/6/

  • Roschelle, J., Tatar, D., & Kaput, J. (2008). Getting to scale with innovations that deeply restructure how students come to know mathematics. In A. E. Kelly, R. Lesh, & J.Y. Baek (Eds.), Handbook of innovative design research in science, technology, engineering, mathematics education. Hillsdale, NJ: Lawrence Erlbaum Associates (in press).

  • Roschelle, J., Tatar, D., Shechtman, N., Hegedus, S., Hopkins, B., & Knudsen, J. (2007). Can a technology-enhanced curriculum improve student learning of important mathematics? Results from 7th grade, year 1 (No. 1). Menlo Park, CA: SRI International.

    Google Scholar 

  • Schneider, B., & McDonald, S. K. (2007). Introduction. In B. Schneider & S.-K. McDonald (Eds.), Scale-up in education (pp. 1–15). Lanham, MD: Rowman & Littlefield.

    Google Scholar 

  • Smith, J. P., diSessa, A. A., & Roschelle, J. (1993). Misconceptions reconceived: A constructivist analysis of knowledge in transition. Journal of the Learning Sciences, 3(2), 115–163.

    Article  Google Scholar 

  • Tatar, D., Roschelle, J., Knudsen, J., Shechtman, N., Kaput, J., & Hopkins, B. (2008). Scaling up innovative technology-based math. Journal of the Learning Sciences (in press).

  • Torgerson, C. (2001). The need for randomised controlled trials in educational research. British Journal of Educational Studies, 49(3), 316–328.

    Article  Google Scholar 

Download references

Acknowledgement

Thank you to our colleagues who helped carry out our scaling up research at SRI International, the University of Massachusetts, Dartmouth, Virginia Tech, the University of Texas, Austin, and the Charles A. Dana Center. We also thank all the teachers and educational service center leaders who participated in this research. This material is based upon work supported by the National Science Foundation under Grant No. REC-0437861. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to J. Roschelle.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Roschelle, J., Tatar, D., Shechtman, N. et al. The role of scaling up research in designing for and evaluating robustness. Educ Stud Math 68, 149–170 (2008). https://doi.org/10.1007/s10649-008-9119-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10649-008-9119-3

Keywords

Navigation