Uwe Naumann
Skip Abstract Section
Abstract
Adjoint methods have become fundamental ingredients of the scientific computing toolbox over the past decades. Large-scale parameter sensitivity analysis, uncertainty quantification, and nonlinear optimization would otherwise turn out computationally infeasible. The symbolic derivation of adjoint mathematical models for relevant problems in science and engineering and their implementation in consistency with the implementation of the underlying primal model frequently proves highly challenging. Hence, an increased interest in algorithmic adjoints can be observed.
The algorithmic derivation of adjoint numerical simulation programs shifts some of the problems faced from functional and numerical analysis to computer science. It becomes a highly complex software engineering task requiring expertise in software analysis, transformation, and optimization. Despite rather mature software tool support for algorithmic differentiation, substantial user intervention is typically required when targeting nontrivial numerical programs. A large number of patterns shared by numerous application codes results in repeated duplication of development effort. The adjoint code design patterns introduced in this article aim to reduce this problem through improved formalization from the software engineering perspective. Fully functional reference implementations are provided through github.
- T. Albring, M. Sagebaum, and N. R. Gauger. 2015. Development of a consistent discrete adjoint solver in an evolving aerodynamic design framework. In Proceedings of the 16th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference (AIAA’15).Google Scholar
- J. Barhen and D. Reister. 2003. Uncertainty analysis based on sensitivities generated using automatic differentiation. In Proceedings of the International Conference on Computational Science and its Applications (ICCSA’03) (Lecture Notes in Computer Science), V. Kumar, M. Gavrilova, C. Tan, and P. L’Ecuyer (Eds.), vol. 2668. Springer, Berlin, 70--77. Google ScholarDigital Library
- A. Baydin, B. Pearlmutter, and A. Radul. 2015. Automatic differentiation in machine learning: A survey. CoRR abs/1502.05767 (2015). Google ScholarDigital Library
- M. Beckers, V. Mosenkis, and U. Naumann. 2012. Adjoint mode computations of subgradients for McCormick relaxations. In Recent Advances in Algorithmic Differentiation (Lecture Notes in Computational Science and Engineering). Springer, 103--113.Google Scholar
- B. Bell and J. Burke. 2008. Algorithmic differentiation of implicit functions and optimal values. In Advances in Automatic Differentiation, C. Bischof, M. Bücker, P. Hovland, U. Naumann, and J. Utke (Eds.). Springer, 67--77.Google Scholar
- M. Berz, C. Bischof, G. Corliss, and A. Griewank (Eds.). 1996. Computational Differentiation: Techniques, Applications and Tools. SIAM, Philadelphia, PA.Google Scholar
- C. Bischof, M. Bücker, P. Hovland, U. Naumann, and J. Utke (Eds.). 2008. Advances in Automatic Differentiation. Number 64 in Lecture Notes in Computational Science and Engineering. Springer. Google ScholarDigital Library
- C. Bischof, A. Carle, P. Khademi, and A. Mauer. 1996. ADIFOR 2.0: Automatic differentiation of Fortran 77 programs. IEEE Comput. Sci. Engineer. 3, 3 (1996), 18--32. Google ScholarDigital Library
- G. Booch. 2004. Object-Oriented Analysis and Design with Applications (3rd ed.). Addison Wesley Longman Publishing Co., Inc., Redwood City, CA. Google ScholarDigital Library
- G. Booch, J. Rumbaugh, and I. Jacobson. 2005. Unified Modeling Language User Guide (2nd ed.) (Addison-Wesley Object Technology Series). Addison-Wesley Professional. Google ScholarDigital Library
- M. Bücker, G. Corliss, P. Hovland, U. Naumann, and B. Norris (Eds.). 2006. Automatic Differentiation: Applications, Theory, and Tools. Number 50 in Lecture Notes in Computational Science and Engineering. Springer. Google ScholarDigital Library
- J. Bunch and J. Hopcroft. 1974. Triangular factorization and inversion by fast matrix multiplication. Math. Comput. 28, 125 (1974), 231--236.Google ScholarCross Ref
- J. Burgers. 1939. Mathematical examples illustrating relations occurring in the theory of turbulent fluid motion. Verhandelingen der Koninklijke Nederlandse Akademie van Wetenschappen, Afdeeling Natuurkunde 2, 17 (1939), 1--53.Google Scholar
- L. Capriotti and M. Giles. 2012. Algorithmic differentiation: Adjoint greeks made easy. Risk.net (2012).Google Scholar
- B. Christianson. 1994. Reverse accumulation and attractive fixed points. Optim. Methods Software 3, 4 (1994), 311--326.Google ScholarCross Ref
- G. Corliss, C. Faure, A. Griewank, L. Hascoët, and U. Naumann (Eds.). 2002. Automatic Differentiation of Algorithms: From Simulation to Optimization. Springer. Google ScholarDigital Library
- M. Fagan and A. Carle. 2005. Reducing reverse-mode memory requirements by using profile-driven checkpointing. Future Gen. Comput. Syst. 21, 8 (2005), 1380--1390. Google ScholarDigital Library
- S. Forth, P. Hovland, E. Phipps, J. Utke, and A. Walther (Eds.). 2012. Recent Advances in Algorithmic Differentiation. Lecture Notes in Computational Science and Engineering, vol. 87. Springer, Berlin. Google ScholarDigital Library
- E. Gamma, R. Helm, R. Johnson, and J. Vlissides. 1995. Design Patterns: Elements of Reusable Object-oriented Software. Addison-Wesley Longman Publishing, Boston, MA. Google ScholarDigital Library
- A. Gebremedhin, F. Manne, and A. Pothen. 2005. What color is your Jacobian? Graph coloring for computing derivatives. SIAM Rev. 47, 4 (2005), 629--705. Google ScholarDigital Library
- R. Giering and T. Kaminski. 1998. Recipes for adjoint code construction. ACM Trans. Math. Software 24, 4 (1998), 437--474. Google ScholarDigital Library
- J. C. Gilbert. 1992. Automatic differentiation and iterative processes. Optim. Methods Software 1 (1992), 13--21.Google ScholarCross Ref
- Mike B. Giles. 2008. Collected matrix derivative results for forward and reverse mode algorithmic differentiation. In Advances in Automatic Differentiation, C. Bischof, M. Bücker, P. Hovland, U. Naumann, and J. Utke (Eds.). Springer, 35--44.Google Scholar
- F. Gremse, A. Hoefter, L. Razik, F. Kiessling, and U. Naumann. 2016. GPU-ACCELERATED adjoint algorithmic differentiation. Comput. Phys. Commun. 200 (2016), 300--311.Google ScholarCross Ref
- A. Griewank. 1992. Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiation. Optim. Methods Software 1 (1992), 35--54.Google ScholarCross Ref
- A. Griewank. 2013. On stable piecewise linearization and generalized algorithmic differentiation. Optim. Methods Software 28, 6 (2013), 1139--1178. Google ScholarDigital Library
- A. Griewank, C. Bischof, G. Corliss, A. Carle, and K. Williamson. 1993. Derivative convergence for iterative equation solvers. Optim. Methods Software 2 (1993), 321--355.Google ScholarCross Ref
- A. Griewank and G. Corliss (Eds.). 1991. Automatic Differentiation of Algorithms: Theory, Implementation, and Application. SIAM, Philadelphia, PA.Google Scholar
- Andreas Griewank and Christèle Faure. 2002. Reduced functions, gradients and Hessians from fixed-point iterations for state equations. Numer. Algor. 30, 2 (2002), 113--139.Google ScholarCross Ref
- A. Griewank, D. Juedes, and J. Utke. 1996. Algorithm 755: ADOL-C: A package for the automatic differentiation of algorithms written in C/C++. ACM Trans. Math. Software 22, 2 (1996), 131--167. Google ScholarDigital Library
- A. Griewank and U. Naumann. 2003. Accumulating Jacobians as chained sparse matrix products. Math. Program. 95, 3 (2003), 555--571.Google ScholarCross Ref
- A. Griewank and A. Walther. 2000. Algorithm 799: Revolve: An implementation of checkpointing for the reverse or adjoint mode of computational differentiation. ACM Trans. Math. Software 26, 1 (Mar. 2000), 19--45. Google ScholarDigital Library
- A. Griewank and A. Walther. 2002. On constrained optimization by adjoint-based quasi-Newton methods. Optim. Methods Software 17 (2002), 869--889.Google ScholarCross Ref
- A. Griewank and A. Walther. 2008. Evaluating Derivatives. Principles and Techniques of Algorithmic Differentiation, Seocnd Edition. Number OT105 in Other Titles in Applied Mathematics. SIAM. Google ScholarDigital Library
- L. Hascoët, U. Naumann, and V. Pascual. 2005. To-be-recorded analysis in reverse mode automatic differentiation. Future Gen. Comput. Syst. 21 (2005), 1401--1417. Google ScholarDigital Library
- L. Hascoët and V. Pascual. 2013. The Tapenade automatic differentiation tool: Principles, model, and specification. ACM Trans. Math. Software 39, 3 (2013), 20:1--20:43. Google ScholarDigital Library
- L. Hascoët, J. Utke, and U. Naumann. 2008. Cheaper adjoints by removing address computations. Sci. Program. 16, 1 (2008), 81--92. Google ScholarDigital Library
- P. Heimbach, C. Hill, and R. Giering. 2005. An efficient exact adjoint of the parallel MIT general circulation model, generated via automatic differentiation. Future Gen. Comput. Syst. 21, 8 (2005), 1356--1371. Google ScholarDigital Library
- M. Hestenes and E. Stiefel. 1952. Methods of conjugate gradients for solving linear systems. J. Res. Nat. Bur. Standards 42, 6 (1952).Google Scholar
- R. Hogan. 2014. Fast reverse-mode automatic differentiation using expression templates in C++. ACM Trans. Math. Software 40, 4 (June 2014), 26:1--26:24. Google ScholarDigital Library
- J. Lotz. 2016. Hybrid Approaches to Adjoint Code Generation with dco/c++. Ph.D. Dissertation. RWTH Aachen University.Google Scholar
- U. Naumann. 2004. Optimal accumulation of Jacobian matrices by elimination methods on the dual computational graph. Math. Program. 99, 3 (2004), 399--421.Google ScholarDigital Library
- U. Naumann. 2009. DAG reversal is NP-complete. J. Discrete Algor. 7 (2009), 402--410. Google ScholarDigital Library
- U. Naumann. 2012. The Art of Differentiating Computer Programs: An Introduction to Algorithmic Differentiation. Number SE24 in Software, Environments, and Tools. SIAM. Google ScholarDigital Library
- U. Naumann and J. du Toit. 2018. Adjoint algorithmic differentiation tool support for typical numerical patterns in computational finance. J. Comput. Finance 21, 4 (2018), 23--57.Google ScholarCross Ref
- U. Naumann, K. Leppkes, and J. Lotz (Eds.). 2014. dco/c++ User Guide. Number AIB-2014-03 in Aachener Informatikberichte. RWTH Aachen University.Google Scholar
- U. Naumann, J. Lotz, K. Leppkes, and M. Towara. 2015. Algorithmic differentiation of numerical methods: Tangent and adjoint solvers for parameterized systems of nonlinear equations. ACM Trans. Math. Software 41, 4 (2015), 26. Google ScholarDigital Library
- A. Noack and A. Walther. 2007. Adjoint concepts for the optimal control of Burgers equation. Comput. Optim. Appl. 36, 1 (2007), 109--133. Google ScholarDigital Library
- C. Noone, M. Torrilhon, and A. Mitsos. 2012. Heliostat field optimization: A new computationally efficient model and biomimetic layout. Solar Energy 86, 2 (2012), 792--803.Google ScholarCross Ref
- G. Ostrowski, Y. Volin, and W. Borisov. 1971. Über die Berechnung von Ableitungen. Wissenschaftliche Zeitschrift der Technischen Hochschule für Chemie, Leuna-Merseburg 13 (1971), 382--384.Google Scholar
- M. Probst, M. Lülfesmann, M. Nicolai, M. Bücker, M. Behr, and C. Bischof. 2010. Sensitivity of optimal shapes of artificial grafts with respect to flow parameters. Comput. Methods Appl. Mech. Engineer. 199, 17--20 (2010), 997--1005.Google Scholar
- M. Sagebaum, T. Albring, and N. R. Gauger. 2018. Expression templates for primal value taping in the reverse mode of algorithmic differentiation. Optim. Methods Software 33, 4--6 (2018), 1207--1231.Google ScholarCross Ref
- P. Stumm and A. Walther. 2008. Multi-stage approaches for optimal offline checkpointing. SIAM journal of scientific computing. SIAM J. Sci. Comput. 31 (2008), 1946--1967. Google ScholarDigital Library
- M. Towara, M. Schanen, and U. Naumann. 2015. MPI-parallel discrete adjoint OpenFOAM. Procedia Comput. Sci. 51 (2015), 19--28. Google ScholarDigital Library
- J. Utke, A. Lyons, and U. Naumann. 2006. Efficient reversal of the intraprocedural flow of control in adjoint computations. J. Syst. Software 79, 9 (2006), 1280--1294. Google ScholarDigital Library
- J. Utke, U. Naumann, M. Fagan, N. Tallent, M. Strout, P. Heimbach, C. Hill, and C. Wunsch. 2008. OpenAD/F: A modular open-source tool for automatic differentiation of Fortran codes. ACM Trans. Math. Software 34, 4 (July 2008). Google ScholarDigital Library
Index Terms
- Adjoint Code Design Patterns
Recommendations
Categorical study for algebras of Fitting’s lattice-valued logic and lattice-valued modal logic
AbstractThe paper explores categorical interconnections between lattice-valued relational systems and algebras of Fitting’s lattice-valued modal logic. We define lattice-valued Boolean systems, and then we study adjointness and co-adjointness of functors ...
Multi-adjoint t-concept lattices
The t-concept lattice is introduced as a set of triples associated to graded tabular information interpreted in a non-commutative fuzzy logic. Following the general techniques of formal concept analysis, and based on the works by Georgescu and Popescu, ...
Multi-adjoint algebras versus non-commutative residuated structures
Multi-adjoint algebras and several properties have been introduced.Adjoint triple and its "dual" cannot be considered in the same framework.A comparison among general residuated algebraic structures has been introduced.Multi-adjoint algebras generalize ...
Comments