Skip to main content
SpringerLink
Log in

Algorithms: A Survey

  • Chapter
  • First Online:
Design of Experiments in Nonlinear Models

Part of the book series: Lecture Notes in Statistics ((LNS,volume 212))

Abstract

We consider the maximization of a design criterion ϕ( ⋅) with respect to ξ ∈ Ξ, the set of probability measures on \(\mathcal{X}\) compact.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    It may happen that the maximum in (9.3) is reached at several x, in particular when the model possesses some symmetry. In that case, a faster convergence is obtained if the multiple maximizers x k + 1, i  +  (i = 1, 2, …, q) are introduced in one single step, using \(\xi _{k+1} = (1 -\alpha _{k})\xi _{k} + (\alpha _{k}/q)\sum _{i}\delta _{x_{ k+1,i}^{+}}\); see Atwood [1973].

  2. 2.

    This is the ellipsoid method used by Khachiyan [1979] to prove the polynomial complexity of LP.

  3. 3.

    There exists a situation, however, where an approximate design can be implemented directly, without requiring any rounding of its weights, whatever their value might be. This is when the design corresponds to the construction of the optimal input signal for a dynamical system, this input signal being characterized by its power spectral density (which plays the same role as the design measure ξ); see, e.g., Goodwin and Payne [1977, Chap. 6], Zarrop [1979], Ljung [1987, Chap. 14], Walter and Pronzato [1997, Chap. 6].

  4. 4.

    The cutting-plane method of Sect. 9.5.3 forms an exception due to the use of an LP solver providing precise solutions; however, it is restricted to moderate values of ℓ; see Examples 9.16 and 9.17.

References

  • Ahipasaoglu, S., P. Sun, and M. Todd (2008). Linear convergence of a modified Frank-Wolfe algorithm for computing minimum volume enclosing ellipsoids. Optim. Methods Softw. 23, 5–19.

    Article  MathSciNet  MATH  Google Scholar 

  • Atwood, C. (1973). Sequences converging to D-optimal designs of experiments. Ann. Statist. 1(2), 342–352.

    Article  MathSciNet  MATH  Google Scholar 

  • Atwood, C. (1976). Convergent design sequences for sufficiently regular optimality criteria. Ann. Statist. 4(6), 1124–1138.

    Article  MathSciNet  MATH  Google Scholar 

  • Atwood, C. (1980). Convergent design sequences for sufficiently regular optimality criteria, II singular case. Ann. Statist. 8(4), 894–912.

    Article  MathSciNet  MATH  Google Scholar 

  • Avriel, M. (2003). Nonlinear Programming. Analysis and Methods. New York: Dover. [Originally published by Prentice Hall, 1976].

    Google Scholar 

  • Ben-Tal, A. and A. Nemirovskii (2001). Lectures on Modern Convex Optimization: Analysis, Algorithms, and Engineering Applications. Philadelphia: MPS/SIAM Series on Optim. 2.

    Google Scholar 

  • Birgin, E., J. Martínez, and M. Raydan (2000). Nonmonotone spectral projected gradient methods on convex sets. SIAM J. Optim. 10(4), 1196–1211.

    Article  MathSciNet  MATH  Google Scholar 

  • Bland, R., D. Goldfarb, and M. Todd (1981). The ellipsoid method: a survey. Oper. Res. 29(6), 1039–1091.

    Article  MathSciNet  MATH  Google Scholar 

  • Bohachevsky, I., M. Johnson, and M. Stein (1986). Generalized simulated annealing for function optimization. Technometrics 28(3), 209–217.

    Article  MATH  Google Scholar 

  • Böhning, D. (1985). Numerical estimation of a probability measure. J. Stat. Plann. Inference 11, 57–69.

    Article  MATH  Google Scholar 

  • Böhning, D. (1986). A vertex-exchange-method in D-optimal design theory. Metrika 33, 337–347.

    Article  MathSciNet  MATH  Google Scholar 

  • Boissonnat, J.-D. and M. Yvinec (1998). Algorithmic Geometry. Cambridge Univ. Press.

    Book  MATH  Google Scholar 

  • Bonnans, J., J. Gilbert, C. Lemaréchal, and C. Sagastizábal (2006). Numerical Optimization. Theoretical and Practical Aspects. Heidelberg: Springer. [2nd ed.].

    Google Scholar 

  • Boyd, S. and L. Vandenberghe (2004). Convex Optimization. Cambridge: Cambridge Univ. Press.

    MATH  Google Scholar 

  • Calamai, P. and J. Moré (1987). Projected gradient methods for linearly constrained problems. Math. Programming 39, 93–116.

    Article  MathSciNet  MATH  Google Scholar 

  • Cook, R. and C. Nachtsheim (1980). A comparison of algorithms for constructing exact D-optimal designs. Technometrics 22(3), 315–324.

    Article  MATH  Google Scholar 

  • Cormen, T., C. Leiserson, R. Rivest, and C. Stein (2001). Introduction to Algorithms. MIT Press and McGraw-Hill. [2nd ed.].

    Google Scholar 

  • Correa, R. and C. Lemaréchal (1993). Convergence of some algorithms for convex minimization. Math. Programming 62, 261–275.

    Article  MathSciNet  MATH  Google Scholar 

  • Dai, Y.-H. and R. Fletcher (2006). New algorithms for singly linearly constrained quadratic programs subject to lower and upper bounds. Math. Programming A 106, 403–421.

    Article  MathSciNet  MATH  Google Scholar 

  • Dem’yanov, V. and V. Malozemov (1974). Introduction to Minimax. New York: Dover.

    Google Scholar 

  • den Boeff, E. and D. den Hertog (2007). Efficient line search methods for convex functions. SIAM J. Optim. 18(1), 338–363.

    Article  MathSciNet  Google Scholar 

  • den Hertog, D. (1994). Interior Point Approach to Linear, Quadratic and Convex Programming. Dordrecht: Kluwer.

    Book  MATH  Google Scholar 

  • Dette, H., A. Pepelyshev, and A. Zhigljavsky (2008). Improving updating rules in multiplicative algorithms for computing D-optimal designs. J. Stat. Plann. Inference 53, 312–320.

    MathSciNet  MATH  Google Scholar 

  • Ecker, J. and M. Kupferschmid (1983). An ellipsoid algorithm for nonlinear programming. Math. Programming 27, 83–106.

    Article  MathSciNet  MATH  Google Scholar 

  • Ecker, J. and M. Kupferschmid (1985). A computational comparison of the ellipsoidal algorithm with several nonlinear programming algorithms. SIAM J. Control Optim. 23(5), 657–674.

    Article  MathSciNet  MATH  Google Scholar 

  • Ermoliev, Y. and R.-B. Wets (Eds.) (1988). Numerical Techniques for Stochastic Optimization Problems. Berlin: Springer.

    Google Scholar 

  • Fedorov, V. (1971). The design of experiments in the multiresponse case. Theory Probab. Appl. 16(2), 323–332.

    Article  MATH  Google Scholar 

  • Fedorov, V. (1972). Theory of Optimal Experiments. New York: Academic Press.

    Google Scholar 

  • Fellman, J. (1974). On the allocation of linear observations. Comment. Phys. Math. 44, 27–78.

    MathSciNet  Google Scholar 

  • Fellman, J. (1989). An empirical study of a class of iterative searches for optimal designs. J. Stat. Plann. Inference 21, 85–92.

    Article  Google Scholar 

  • Frank, M. and P. Wolfe (1956). An algorithm for quadratic programming. Naval Res. Logist. Quart. 3, 95–110.

    Article  MathSciNet  Google Scholar 

  • Galil, Z. and J. Kiefer (1980). Time- and space-saving computer methods, related to Mitchell’s DETMAX, for finding D-optimum designs. Technometrics 22(3), 301–313.

    Article  MathSciNet  MATH  Google Scholar 

  • Gauchi, J.-P. and A. Pázman (2006). Designs in nonlinear regression by stochastic minimization of functionals of the mean square error matrix. J. Stat. Plann. Inference 136, 1135–1152.

    Article  MATH  Google Scholar 

  • Goffin, J.-L. and K. Kiwiel (1999). Convergence of a simple subgradient level method. Math. Programming 85, 207–211.

    Article  MathSciNet  MATH  Google Scholar 

  • Goodwin, G. and R. Payne (1977). Dynamic System Identification: Experiment Design and Data Analysis. New York: Academic Press.

    MATH  Google Scholar 

  • Grötschel, M., L. Lovász, and A. Schrijver (1980). Geometric Algorithms and Combinatorial Optimization. Berlin: Springer.

    Google Scholar 

  • Harman, R. and L. Pronzato (2007). Improvements on removing non-optimal support points in D-optimum design algorithms. Statist. Probab. Lett. 77, 90–94.

    Article  MathSciNet  MATH  Google Scholar 

  • Hearn, D. and S. Lawphongpanich (1989). Lagrangian dual ascent by generalized linear programming. Oper. Research Lett. 8, 189–196.

    Article  MathSciNet  MATH  Google Scholar 

  • Hiriart-Urruty, J. and C. Lemaréchal (1993). Convex Analysis and Minimization Algorithms, part 1 and 2. Berlin: Springer.

    Google Scholar 

  • Johnson, M. and C. Nachtsheim (1983). Some guidelines for constructing exact D-optimal designs on convex design spaces. Technometrics 25, 271–277.

    MathSciNet  MATH  Google Scholar 

  • Kelley, J. (1960). The cutting plane method for solving convex programs. SIAM J. 8, 703–712.

    MathSciNet  Google Scholar 

  • Khachiyan, L. (1979). A polynomial algorithm in linear programming. Doklady Akademïa Nauk SSSR 244, 1093–1096. [English transl. Soviet Math. Doklady, 20, 191–194].

    Google Scholar 

  • Khachiyan, L. (1996). Rounding of polytopes in the real number model of computation. Math. Oper. Res. 21(2), 307–320.

    Article  MathSciNet  MATH  Google Scholar 

  • Khachiyan, L. and M. Todd (1993). On the complexity of approximating the maximal inscribed ellipsoid for a polytope. Math. Programming A61(2), 137–159.

    Google Scholar 

  • Kushner, H. and D. Clark (1978). Stochastic Approximation Methods for Constrained and Unconstrained Systems. Heidelberg: Springer.

    Book  Google Scholar 

  • Kushner, H. and J. Yang (1993). Stochastic approximation with averaging of the iterates: optimal asymptotic rate of convergence for general processes. SIAM J. Control Optim. 31(4), 1045–1062.

    Article  MathSciNet  MATH  Google Scholar 

  • Kushner, H. and G. Yin (1997). Stochastic Approximation Algorithms and Applications. Heidelberg: Springer.

    MATH  Google Scholar 

  • Lemaréchal, C., A. Nemirovskii, and Y. Nesterov (1995). New variants of bundle methods. Math. Programming 69(1), 111–147.

    Article  MathSciNet  MATH  Google Scholar 

  • Levin, A. (1965). On an algorithm for the minimization of convex functions. Soviet Math. Dokl. 6, 286–290.

    Google Scholar 

  • Ljung, L. (1987). System Identification, Theory for the User. Englewood Cliffs: Prentice Hall.

    MATH  Google Scholar 

  • McCormick, G. and R. Tapia (1972). The gradient projection method under mild differentiability conditions. SIAM J. Control 10(1), 93–98.

    Article  MathSciNet  MATH  Google Scholar 

  • Minoux, M. (1983). Programmation Mathématique, Théorie et Algorithmes, vol. 1 & 2. Paris: Dunod.

    Google Scholar 

  • Molchanov, I. and S. Zuyev (2001). Variational calculus in the space of measures and optimal design. In A. Atkinson, B. Bogacka, and A. Zhigljavsky (Eds.), Optimum Design 2000, Chapter 8, pp. 79–90. Dordrecht: Kluwer.

    Google Scholar 

  • Molchanov, I. and S. Zuyev (2002). Steepest descent algorithm in a space of measures. Stat. Comput. 12, 115–123.

    Article  MathSciNet  Google Scholar 

  • Nesterov, Y. (1995). Complexity estimates of some cutting plane methods based on the analytic center. Math. Programming 69, 149–176.

    MathSciNet  MATH  Google Scholar 

  • Nesterov, Y. (2004). Introductory Lectures to Convex Optimization: A Basic Course. Dordrecht: Kluwer.

    Google Scholar 

  • Nesterov, Y. and A. Nemirovskii (1994). Interior-Point Polynomial Algorithms in Convex Programming. Philadelphia: SIAM.

    Book  MATH  Google Scholar 

  • Pázman, A. (1986). Foundations of Optimum Experimental Design. Dordrecht (co-pub. VEDA, Bratislava): Reidel (Kluwer group).

    Google Scholar 

  • Pázman, A. and L. Pronzato (1992). Nonlinear experimental design based on the distribution of estimators. J. Stat. Plann. Inference 33, 385–402.

    Article  MATH  Google Scholar 

  • Polak, E. (1971). Computational Methods in Optimization, a Unified Approach. New York: Academic Press.

    Google Scholar 

  • Polyak, B. (1987). Introduction to Optimization. New York: Optimization Software.

    Google Scholar 

  • Polyak, B. (1990). New stochastic approximation type procedures. Automat. i Telemekh. 7, 98–107.

    MathSciNet  Google Scholar 

  • Polyak, B. and A. Juditsky (1992). Acceleration of stochastic approximation by averaging. SIAM J. Control Optim. 30, 838–855.

    Article  MathSciNet  MATH  Google Scholar 

  • Pronzato, L. and E. Walter (1985). Robust experiment design via stochastic approximation. Math. Biosci. 75, 103–120.

    Article  MathSciNet  MATH  Google Scholar 

  • Pronzato, L. and E. Walter (1988). Robust experiment design via maximin optimization. Math. Biosci. 89, 161–176.

    Article  MathSciNet  MATH  Google Scholar 

  • Pronzato, L., H. Wynn, and A. Zhigljavsky (2000). Dynamical Search. Boca Raton: Chapman & Hall/CRC.

    MATH  Google Scholar 

  • Pukelsheim, F. and S. Reider (1992). Efficient rounding of approximate designs. Biometrika 79(4), 763–770.

    Article  MathSciNet  Google Scholar 

  • Robertazzi, T. and S. Schwartz (1989). An accelerated sequential algorithm for producing D-optimal designs. SIAM J. Sci. Statist. Comput. 10(2), 341–358.

    Article  MathSciNet  Google Scholar 

  • Shah, S., J. Mitchell, and M. Kupferschmid (2000). An ellipsoid algorithm for equality-constrained nonlinear programs. Comput. Oper. Res. 28(1), 85–92.

    Article  MathSciNet  Google Scholar 

  • Shimizu, K. and E. Aiyoshi (1980). Necessary conditions for min-max problems and algorithm by a relaxation procedure. IEEE Trans. Automatic Control 25, 62–66.

    Article  MathSciNet  MATH  Google Scholar 

  • Shor, N. (1977). Cut-off method with space extension in convex programming problems. Kibernetika 13(1), 94–95. [English Trans. Cybernetics Syst. Anal., 13(1), 94–96].

    Google Scholar 

  • Shor, N. (1985). Minimization Methods for Non-Differentiable Functions. Berlin: Springer.

    Book  MATH  Google Scholar 

  • Shor, N. and O. Berezovski (1992). New algorithms for constructing optimal circumscribed and inscribed ellipsoids. Optim. Methods Softw. 1, 283–299.

    Article  Google Scholar 

  • Silvey, S., D. Titterington, and B. Torsney (1978). An algorithm for optimal designs on a finite design space. Comm. Statist. – Theory Methods A7(14), 1379–1389.

    Google Scholar 

  • St. John, R. and N. Draper (1975). D-optimality for regression designs: a review. Technometrics 17(1), 15–23.

    Google Scholar 

  • Tarasov, S., L. Khachiyan, and I. Erlich (1988). The method of inscribed ellipsoids. Soviet Math. Dokl. 37(1), 226–230.

    MathSciNet  MATH  Google Scholar 

  • Titterington, D. (1976). Algorithms for computing D-optimal designs on a finite design space. In Proc. 1976 Conf. on Information Science and Systems, Baltimore, pp. 213–216. Dept. of Electronic Engineering, John Hopkins Univ.

    Google Scholar 

  • Titterington, D. (1978). Estimation of correlation coefficients by ellipsoidal trimming. J. Roy. Statist. Soc. C27(3), 227–234.

    Google Scholar 

  • Todd, M. (1982). On minimum volume ellipsoids containing part of a given ellipsoid. Math. Oper. Res. 7(2), 253–261.

    Article  MathSciNet  MATH  Google Scholar 

  • Todd, M. and E. Yildirim (2007). On Khachiyan’s algorithm for the computation of minimum volume enclosing ellipsoids. Discrete Appl. Math. 155, 1731–1744.

    Article  MathSciNet  MATH  Google Scholar 

  • Torsney, B. (1983). A moment inequality and monotonicity of an algorithm. In K. Kortanek and A. Fiacco (Eds.), Proc. Int. Symp. on Semi-infinite Programming and Applications, Heidelberg, pp. 249–260. Springer.

    Google Scholar 

  • Torsney, B. (2009). W-iterations and ripples therefrom. In L. Pronzato and A. Zhigljavsky (Eds.), Optimal Design and Related Areas in Optimization and Statistics, Chapter 1, pp. 1–12. Springer.

    Google Scholar 

  • Uciński, D. and M. Patan (1982). D-optimal design of a monitoring network for parameter estimation of distributed systems. J. Global Optim. 39, 291–322.

    Article  Google Scholar 

  • Veinott, A. (1967). The supporting hyperplane method for unimodal programming. Oper. Research 15, 147–152.

    Article  MathSciNet  MATH  Google Scholar 

  • Walter, E. and L. Pronzato (1997). Identification of Parametric Models from Experimental Data. Heidelberg: Springer.

    MATH  Google Scholar 

  • Welch, W. (1982). Branch-and-bound search for experimental designs based on D-optimality and other criteria. Technometrics 24(1), 41–28.

    MathSciNet  MATH  Google Scholar 

  • Wolfe, P. (1970). Convergence theory in nonlinear programming. In J. Abadie (Ed.), Integer and Nonlinear Programming, pp. 1–36. Amsterdam: North Holland.

    Google Scholar 

  • Wright, M. (1998). The interior-point revolution in constrained optimization. Technical Report 98–4–09, Computing Sciences Research Center, Bell Laboratories, Murray Hill, New Jersey 07974.

    Google Scholar 

  • Wu, C. (1978a). Some algorithmic aspects of the theory of optimal designs. Ann. Statist. 6(6), 1286–1301.

    Article  MathSciNet  MATH  Google Scholar 

  • Wu, C. (1978b). Some iterative procedures for generating nonsingular optimal designs. Comm. Statist. – Theory Methods A7(14), 1399–1412.

    Google Scholar 

  • Wu, C. and H. Wynn (1978). The convergence of general step–length algorithms for regular optimum design criteria. Ann. Statist. 6(6), 1273–1285.

    Article  MathSciNet  MATH  Google Scholar 

  • Wynn, H. (1970). The sequential generation of D-optimum experimental designs. Ann. Math. Statist. 41, 1655–1664.

    Article  MathSciNet  MATH  Google Scholar 

  • Wynn, H. (1972). Results in the theory and construction of D-optimum experimental designs. J. Roy. Statist. Soc. B34, 133–147.

    Google Scholar 

  • Ye, Y. (1997). Interior-Point Algorithms: Theory and Analysis. Chichester: Wiley.

    Book  MATH  Google Scholar 

  • Yu, Y. (2010a). Monotonic convergence of a general algorithm for computing optimal designs. Ann. Statist. 38(3), 1593–1606.

    Article  MathSciNet  MATH  Google Scholar 

  • Yu, Y. (2010b). Strict monotonicity and convergence rate of Titterington’s algorithm for computing D-optimal designs. Comput. Statist. Data Anal. 54, 1419–1425.

    Article  MathSciNet  Google Scholar 

  • Yu, Y. (2011). D-optimal designs via a cocktail algorithm. Stat. Comput. 21, 475–481.

    Article  MathSciNet  MATH  Google Scholar 

  • Zarrop, M. (1979). Optimal Experiment Design for Dynamic System Identification. Heidelberg: Springer.

    Book  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. French National Center for Scientific Research (CNRS), University of Nice, Sophia Antipolis, France

    Luc Pronzato

  2. Department of Applied Mathematics and Statistics, Comenius University, Bratislava, Slovakia

    Andrej Pázman

Authors
  1. Luc Pronzato
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrej Pázman
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer Science+Business Media New York

About this chapter

Cite this chapter

Pronzato, L., Pázman, A. (2013). Algorithms: A Survey. In: Design of Experiments in Nonlinear Models. Lecture Notes in Statistics, vol 212. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-6363-4_9

Download citation

Publish with us

Policies and ethics

Search

Navigation