Constrained distributed optimization: A population dynamics approach☆
Introduction
An approach to design control systems is to express the desired performance of the plant as an optimization problem with multiple constraints, e.g., minimization of the error, minimization of the norm of states, minimization of the energy associated to control actions, all of those objectives subject to physical and/or operational constraints. When the system involves a large number of states, the design of optimization-based controllers becomes challenging, because of the lack of centralized information or because of other implications associated to information (e.g., communication issues, costs, reliability). The limitation regarding information availability demands the development of distributed optimization techniques that achieve an optimal point of a performance cost function for the total system by using only local and partial information. There are many distributed optimization applications in engineering, and most of them using a network systems approach (Bertsekas, 2012, Gao and Cheng, 2005, Simonetto et al., 2010, Simonetto et al., 2011). These problems have been solved by using distributed optimization algorithms based on the Newton method (Jadbabaie et al., 2009, Wei et al., 2013), the sub-gradient method (Johansson et al., 2008, Zhu and Martinez, 2012), and the consensus protocol (Johansson et al., 2008, Notarstefano and Bullo, 2011, Zhang and Liu, 2014), among other techniques. On the other hand, game theory studies the interaction of decision makers and the interconnection of decision making elements based on local information. From this perspective, game-theoretical tools become very useful to describe the behavior of distributed engineered systems (Marden & Shamma, 2015). One important characteristic of this theoretical approach is the Nash equilibrium concept, which describes how a global objective is reached based only on local decisions. The task to reach a global objective with partial information is one of the main aspects in distributed optimization problems. This problem may be seen as a multi-agent case in which there are local interactions among them. Furthermore, evolutionary game theory describes the previously mentioned model of agents interacting but also considering a determined population structure, i.e., constraints in the interaction among agents, (Nowak, 2006). From this point of view, this theory is suitable to design intelligent systems and controllers for systems where there are local decision makers (local controllers) and achieving a global performance and/or global goal under a specific structure, which is given by the topology of the system (e.g., energy networks, water networks, transportation networks, etc.). Also, game theory has become an important and powerful tool for solving optimization problems since the Nash equilibrium corresponds to the extreme of a potential function satisfying the Karush–Kuhn–Tucker (KKT) first order condition (Sandholm, 2010). This property is commonly used in a class of games known as potential games, which have gotten special importance in the solution of engineering problems. For instance, in Marden (2012) potential games are widely studied from the perspective of state-based games. Furthermore, some kind of optimization problems can be solved by finding a Nash equilibrium for an appropriate designed game, and the consideration of only local information allows to solve distributed optimization problems (Arslan & Shamma, 2004). For instance, in Gharesifard and Cortes (2013) a distributed convergence to Nash equilibria in two networks is discussed for zero-sum games. In Pantoja and Quijano (2011), distributed optimization has been applied using replicator dynamics (one of the six fundamental population dynamics), based on local information. In Li and Marden (2014), the design of utility functions for each agent in order to decouple constraints is presented, and the usage of penalty functions and barrier functions is discussed. The design of local control laws for individual agents to achieve a global objective is proposed in Li and Marden (2013), which has been extended in Zhang, Qi, and Zhao (2013) by using matrix theory. The consideration of dynamics in the system-equivalent graph that describes information sharing among decision variables is paramount since some network systems in engineering might grow (e.g., drainage network systems, drinking water networks, distributed generation systems). These dynamics represent an addition or removal of elements to/from the network. Moreover, the connectivity of the network elements could change over time (e.g., re-configuration systems), which could affect availability of information. In Li and Marden (2012), variations on the graph that determines the system information sharing are studied, where the set of communication links varies with a certain probability.
The main contribution of this paper is to introduce a novel methodology to solve constrained optimization problems in a distributed way, inspired by the population dynamics studied in Sandholm (2010). Different from the already published population dynamics approaches, this method adds dynamics to the population masses, making the population simplex vary properly over time making the method robust (Barreiro-Gomez, Quijano, & Ocampo-Martinez, 2014a). The method consists in considering the global problem as a society, where there is limitation of information sharing. The society is divided into several populations, where there is full available information. Then, a local optimization problem is solved at each population whose feasible region varies dynamically, i.e., there is an interchange of masses among populations. The feasible regions vary until all populations agree to solve the global optimization problem. In addition to this, applications in the control field may involve disturbances that could lead the trajectories to leave the feasible region (given by constraints that impose a desire performance) (Barreiro-Gomez, Quijano, & Ocampo-Martinez, 2014b). Another relevant difference with respect to already published distributed population dynamics approaches is that the proposed method guarantees that the feasible region is attractive. The last mentioned feature potentially improve the control performance rejecting disturbances. Finally, the design of the decision-making distributed system allows to have a reduced number of modifications when the graph topology changes, i.e., there are new nodes/edges in the graph or there are nodes/edges that disappear. Also, some redundant links can be identified, i.e., links in the graph that are not necessary in the connection among cliques.
The remainder of the paper is organized as follows. Section 2 shows preliminaries of graphs, population dynamics, and introduces the mathematical formalism that is used throughout the paper. Section 3 presents the population dynamics and the mass dynamics, including relevant characteristics. Then, the stability analysis of the dynamics is presented in Section 4. Section 5 shows the different possible changes that the social graph might suffer, and explains the implication over the design. Section 6 presents the optimization problem forms that could be solved with the population dynamics and the mass dynamics, presenting also some illustrative examples and results. Afterwards, the robustness of the method is shown by applying disturbances in Section 7. Then, Section 8 presents a large-scale system and the design of optimal controllers by using the proposed methodology. Controllers consider both a model-based approach, and a model-free approach. Section 9 shows the results and discussion about the performance of controllers designed with the proposed methodology. In Section 10 the main conclusions are drawn. Notation The sub-index is associated to a node of a graph, or to a strategy in a game. On the other hand, the super-index refers to a population. For instance, the sub-index in , , or refers either to a node in a graph or to a strategy, and the super-index in , , or indicates a population. Also it should be clear that the super-index is not an operational number, i.e., refers to population three but . We use bold font for column vectors and matrices, e.g., , and ; and non-bold style is used for scalar numbers, e.g., . Calligraphy style is used for sets, e.g., . The column vector with unitary entries is denoted by , and the column vector with null entries and suitable dimension is denoted by . The identity matrix with dimension is denoted by . The cardinality of a set is denoted by . The continuous time is denoted by , and it is mostly omitted throughout the manuscript in order to simplify the notation. Finally, represents the set of all non-negative real numbers, and represents the set of positive integer numbers.
Section snippets
Preliminaries
Let be an undirected non-complete connected graph that exhibits the topology of a society, where is the set of vertices of that represents the set of available strategies in a social game denoted by ; and is the set of edges of that determines the possible interactions among society strategies. The graph is divided into sub-complete graphs known as cliques (a complete sub-graph), where each clique represents a population within the society. The set
Population and mass dynamics
The objective for the society is to converge to a Nash equilibrium1 of the game denoted by . In order to achieve this objective, there is a game at each population converging to a Nash equilibrium of the game denoted by , and the intersection nodes
Stability analysis
It is necessary to show that the solution of the distributed system with population dynamics (6a), and mass dynamics (7a) at intersection nodes, implies the solution of the social game (i.e., the global problem). Proposition 2 If Assumption 3 is satisfied, the population dynamics (6a) are in equilibrium , and the mass dynamics (7a) are in equilibrium ; then, the society is in equilibrium, i.e., .
Proof The equilibrium of the population dynamics (6a), for
Changes in the graph
Network systems in engineering are constantly growing. On the other hand, some systems isolate a segment of the network under some specific conditions (e.g., an operation fault), and this might suppose a reduction in the graph. When any of these two situations occur, the controller should be re-designed in order to fit the new system conditions. The method proposed in this paper allows to reduce the number of changes in the original design when changes in the graph are made. It is necessary to
Optimization problems with constraints
One of the main features of full potential games is that their Nash equilibrium points coincide with the extreme points of the corresponding potential function, i.e., Nash equilibria satisfy the KKT first-order conditions (Sandholm, 2010). Additionally, if the potential function is concave, potential games are stable and an optimization problem can be solved in a distributed way by using the population dynamics and the mass dynamics shown in Section 3. Some optimization problem forms are set
Performance against disturbances and the convergence factor
As it has been shown in Section 3, the convergence factor forces trajectories towards the feasible region. This fact makes the proposed methodology robust against disturbances. In order to illustrate this feature, Example 6.1.1 is solved under different conditions. First, the initial condition does not belong to the feasible region (i.e., ). Additionally, as a second factor affecting the evolution of states, a hard disturbance is applied to a node in the graph shown in Fig. 5. The
Case study
In an optimal control design, the control actions are commonly the decision variables. Then, from the point of view of our proposed approach, the proportion of agents playing each strategy is associated to each control action of the system. It is guaranteed that the solution obtained with the proposed approach satisfies the established constraints. However, due to the fact that trajectories of proportion of agents might get out from the feasible region in a transitory event, it is not
Results and discussion
In order to analyze the proposed methodology, four different scenarios are proposed, two of them designed with the proposed population and mass dynamics approach and two of them by using a centralized MPC approach:
- •
Scenario 1: controller based on the proposed population and mass dynamics, and considering a COM.
- •
Scenario 2: model-free controller based on the proposed population and mass dynamics.
- •
Scenario 3: MPC controller with perfect COM (i.e., the COM equal to the SOM) and with a prediction
Concluding remarks
A methodology to solve different optimization problems with multiple constraints has been presented. The method is based on population dynamics, whose set of states varies over time (i.e., dynamics over the population masses are added). The variation of the set of possible states represents a mass interchange among populations. It has been shown that the population dynamics and the mass dynamics are stable and that the feasible region of the global problem is attractive, under the presented
Acknowledgment
We would like to thank Germán Obando for his inputs and the academic discussions about this work.
Julian Barreiro-Gomez received his B.S. degree in Electronics Engineering from Universidad Santo Tomas (USTA), Bogotá, Colombia, in 2011. He received the M.Sc. degree in Electrical Engineering from Universidad de Los Andes (UAndes), Bogotá, Colombia, in 2013. Since 2012, he is with the research group in control and automation systems (GIAP, UAndes), where he is pursuing the Ph.D. degree in Engineering in the area of control systems. Since 2014, he is an associate researcher at Technical
References (31)
- et al.
Distributed convergence to Nash equilibria in two-network zero-sum games
Automatica
(2013) State based potential games
Automatica
(2012)- et al.
Handbook of game theory with economic applications
- et al.
Partitioning approach oriented to the decentralised predictive control of large-scale systems
Journal of Process Control
(2011) - et al.
Consensus of second-order multi-agent systems with disturbances generated by non-linear exosystems under switching topologies
Journal of the Franklin Institute
(2014) - Arslan, G., & Shamma, J. (2004). Distributed convergence to Nash equilibria with local utility measurements. In...
- Barreiro-Gomez, J., Quijano, N., & Ocampo-Martinez, C. (2014a). Constrained distributed optimization based on...
- Barreiro-Gomez, J., Quijano, N., & Ocampo-Martinez, C. (2014b). Distributed control of drinking water networks using...
Incremental gradient, subgradient and proximal methods for convex optimization: a survey
- et al.
Comment on coordination of groups of mobile autonomous agents using nearest neighbor rules
IEEE Transactions on Automatic Control
(2005)
Stable games and their dynamics
Journal of Economic Theory
Cliques of a graph-variations on the Bron-Kerbosch algorithm
International Journal of Parallel Programming
Cited by (44)
Model Predictive Control of water resources systems: A review and research agenda
2023, Annual Reviews in ControlHierarchical distributed optimization of constraint-coupled convex and mixed-integer programs using approximations of the dual function
2023, EURO Journal on Computational OptimizationConvergence analysis of distributed population dynamics based on second-order Delaunay triangulation
2022, Information SciencesCitation Excerpt :Among various distributed population dynamics, distributed Smith dynamics have been intensively investigated for the capability to properly handle the non-negativity constraints due to its physical nature in various practical problems [27]. Most studies pay attention to modeling the distributed population dynamics for a given communication network or satisfying specific communication networks [47,1,16]. Yet the relationship between communication networks and the convergence of distributed population dynamics is not entirely understood.
Nash equilibrium seeking in full-potential population games under capacity and migration constraints
2022, AutomaticaCitation Excerpt :Regarding GNE seeking problems in evolutionary games, on the other hand, different approaches have been recently proposed. Namely, Barreiro-Gomez, Quijano, and Ocampo-Martinez (2016) consider the migration-constrained population games of Barreiro-Gomez et al. (2017) in the context of density-dependent population games (i.e., population games where the total mass of players is not constant), and propose a primal–dual approach to include affine constraints over the strategic distribution of the population. The proposed approach is illustrated on distributed GNE seeking problems in water distribution systems.
A payoff dynamics model for generalized Nash equilibrium seeking in population games
2022, AutomaticaCitation Excerpt :Some recent approaches that shed light onto this topic are the ones in Barreiro-Gomez et al. (2016), Barreiro-Gomez and Tembine (2018), and Martinez-Piazuelo et al. (2022). Namely, Barreiro-Gomez et al. (2016) introduce the concept of mass dynamics to consider affine constraints in stable population games under certain imitative revision protocol; Barreiro-Gomez and Tembine (2018) propose some novel revision protocols to consider affine box inequality constraints in stable games; and Martinez-Piazuelo et al. (2022) propose some dynamic payoff mechanism for GNE seeking in potential population games with affine equality constraints and under impartial pairwise comparison revision protocols. In contrast with the aforementioned previous works, in this paper we consider the problem of GNE seeking in population games under fairly general constraints.
A payoff-based learning approach for Nash equilibrium seeking in continuous potential games
2022, NeurocomputingCitation Excerpt :In this case, the only available information of each agent is its own past actions and corresponding utilities. The lack of full information may be encountered in a variety of scenarios [20–24]. For instance, in motion coordination problems [25–27], agents may have an ability to sense its distance with other agents, but they may not have an ability to sense other agents’ relative or absolute positions.
Julian Barreiro-Gomez received his B.S. degree in Electronics Engineering from Universidad Santo Tomas (USTA), Bogotá, Colombia, in 2011. He received the M.Sc. degree in Electrical Engineering from Universidad de Los Andes (UAndes), Bogotá, Colombia, in 2013. Since 2012, he is with the research group in control and automation systems (GIAP, UAndes), where he is pursuing the Ph.D. degree in Engineering in the area of control systems. Since 2014, he is an associate researcher at Technical University of Catalonia (Barcelona, Spain), Automatic Control Department (ESAII), and Institut de Robòtica i Informàtica Industrial (CSIC–UPC), where he is pursuing the Ph.D. degree in Automatic, Robotics and Computer Vision. His main research interests are constrained model predictive control, distributed optimization and control, game theory, and population dynamics.
Nicanor Quijano received his B.S. degree in Electronics Engineering from Pontificia Universidad Javeriana (PUJ), Bogotá, Colombia, in 1999. He received the M.S. and Ph.D. degrees in Electrical and Computer Engineering from The Ohio State University, in 2002 and 2006, respectively. In 2007 he joined the Electrical and Electronics Engineering Department, Universidad de los Andes (UAndes), Bogotá, Colombia as Assistant Professor. In 2008 he obtained the Distinguished Lecturer Award from the School of Engineering, UAndes. He is currently Full Professor and the director of the research group in control and automation systems (GIAP, UAndes). On the other hand, he has been a member of the Board of Governors of the IEEE Control Systems Society (CSS) for the 2014 period, and he was the Chair of the IEEE CSS, Colombia for the 2011–2013 period. Currently his research interests include: hierarchical and distributed optimization methods using bio-inspired and game-theoretical techniques for dynamic resource allocation problems, especially those in energy, water, and transportation. For more information and a complete list of publications see: https://profesores.uniandes.edu.co/nquijano/.
Carlos Ocampo-Martinez received his electronics engineering degree and his M.Sc. degree in industrial automation from the National University of Colombia, Campus Manizales, in 2001 and 2003, respectively. In 2007, he received his Ph.D. degree in Control Engineering from the Technical University of Catalonia (Barcelona, Spain). In 2007–2008, he held a postdoctoral position at the ARC Centre of Complex Dynamic Systems and Control (University of Newcastle, Australia) and, afterwards at the Spanish National Research Council (CSIC), Institut de Robòtica i Informàtica Industrial, CSIC–UPC (Barcelona) as a Juan de la Cierva research fellow between 2008 and 2011. Since 2011, he is with the Technical University of Catalonia, Automatic Control Department (ESAII) as Associate Professor in automatic control and model predictive control. Since 2014, he is also Deputy Director of the Institut de Robòtica i Informàtica Industrial (CSIC–UPC), a Joint Research Center of UPC and CSIC. His main research interests include constrained model predictive control, large-scale systems management (partitioning and non-centralized control), and industrial applications (mainly related to the key scopes of water and energy).
- ☆
This work has been partially supported by the projects “Drenaje urbano y cambio climático: hacia los sistemas de alcantarillado del futuro. Colciencias 548/2012”, and ECOCIS (Ref. DPI2013-48243-C2-1-R). Julian Barreiro-Gomez is supported by COLCIENCIAS-COLFUTURO (grant 6172) and by the Agència de Gestió d’Ajust Universitaris i de Recerca AGAUR (grant FI-2014). The material in this paper was partially presented at the 53rd Conference on Decision and Control, December 15–17, 2014, Los Angeles, CA, USA. This paper was recommended for publication in revised form by Editor Berç Rüstem.