An evaluation of multi-model self-managing control schemes for adaptive performance management of software systems

https://doi.org/10.1016/j.jss.2012.05.077Get rights and content

Abstract

Due to the increasing complexity of software systems and the dynamic unpredictable environments they operate in, methodologies to incorporate self-adaptation into these systems have been investigated in recent years. The feedback control loop has been one of the key concepts used in building self-adaptive software systems to manage their performance among other quality aspects. In order to design an effective feedback control loop for a software system, modeling the behavior of the software system with sufficient accuracy is paramount. In general, there are many environmental conditions and system states that impact on the performance of a software system. As a consequence, it is impractical to characterize the diverse behavior of such a software system using a single system model. To represent such highly nonlinear behavior and to provide effective runtime control, the design, integration and self-management (automatic switching) of multiple system models and controllers are required.

In this paper, we investigate a control engineering approach, called Multi-Model Switching and Tuning (MMST) adaptive control, to assess its effectiveness for the adaptive performance management of software systems. We have conducted a range of experiments with two of the most promising MMST adaptive control schemes under different operating conditions of a representative software system. The experiment results have shown that the MMST control schemes are superior in managing the performance of the software system, compared with a number of other control schemes based on a single model. We have also investigated the impact of the configuration parameters for the MMST schemes to provide design guidance. A library of MMST schemes has been implemented to aid the software engineer in developing MMST-based self-managing control schemes for software systems.

Introduction

The growing complexity and criticality of software applications in business operations and the dynamic unpredictable nature of their operating environments demand effective runtime management in order to achieve the required performance objectives and reduce administration costs. The traditional methods that are widely used to manage software systems at runtime, such as manual tuning, have proven to be costly and error-prone (Diao et al., 2003, Salehie and Tahvildari, 2009). The fixed and ad hoc threshold based management policies designed to cope with peak demands are difficult to design due to the lack of a formal or systematic design process (Diao et al., 2003, Dutreilh et al., 2010). To achieve effective runtime performance management for software systems, autonomous or self-adaptive control is required. Such self-adaptive control can be categorized into parameter tuning and architectural reconfiguration (Salehie and Tahvildari, 2009, McKinley et al., 2004). In parameter tuning, a set of predefined system parameters (tuning knobs) are adjusted to maintain the required performance metrics, under dynamic and unpredictable environmental changes (e.g., Hellerstein et al., 2004, Lu et al., 2006, Kandasamy et al., 2004). Architectural reconfigurations, on the other hand, change the system structure and its elements in order to achieve the performance objectives under environmental changes (e.g., Garlan et al., 2004, Sykes et al., 2008). Performing such reconfigurations at runtime can be more costly than parameter tuning (Salehie and Tahvildari, 2009).

Control engineering techniques which utilize the feedback control loop have been applied successfully for decades to achieve control objectives of physical plants. While the feedback control loop has also been identified as one of the enabling techniques to incorporate parameter tuning and architectural reconfiguration decision making capabilities into self-adaptive software systems (Salehie and Tahvildari, 2009, Brun et al., 2009), control engineering techniques have not been widely adopted in software engineering practice (Zhu et al., 2009, Kokar et al., 1999). To effectively apply control engineering techniques to software performance management via parameter tuning in particular, a number of challenges need to be addressed (Brun et al., 2009, Zhu et al., 2009, Hellerstein, 2004). They include the inherent nonlinear behavior of software systems, the need for rigorous mathematical analysis in control system design, and the consequent difficulties in developing accurate models to characterize the system behavior in order to design a controller to manage the system performance at runtime.

Building an accurate model to capture the behavior of a software system has proved to be problematic. First, a complex software system is typically composed of a number of layers, including hardware, operating systems, execution platforms and the application itself. The system performance at the application level is affected by the dynamic and unpredictable behavior of these underlying layers (e.g., garbage collection, compiler optimizations and memory management). Second, the workload conditions at the application level, which are also highly dynamic and unpredictable, affect the system performance as well. For instance, an e-commerce system may face sudden intensive workloads when promotional offers are run or when referenced by a high-traffic site (the so-called ‘slash-dot’ effect). The workloads may also vary dramatically depending on the time of day (e.g., stock market applications) or the time of year (e.g., tax office sites). Third, if the application evolves due to new feature additions, bug fixes or system configuration changes, the constructed model has to be changed as well for accurate representation. The resultant behavior of the software application therefore depends on a combination of the operating states or conditions of the underlying software layers, the current environmental workloads, and the current configuration of the application architecture. Depending on what combination of the above factors occurs, the behavior of the software application can be characterized by a number of distinct operating regions. However, the complexity of interactions between these factors means that creating a single model of the system is very difficult if not virtually impossible.

Existing research efforts that investigate the use of control engineering approaches for software performance management tend to model the system behavior by focusing on a particular operating region under certain operating conditions that can be characterized using a single linear model. Such linearization is inherently problematic because a software system has to work in a spectrum of operating regions with changing conditions and un-modeled system dynamics. In these different operating regions, different control regimes may be needed to achieve the performance objectives effectively. This observation is further confirmed by various studies in the existing literature. For instance, two linear models were implemented in Karlsson et al. (2005b) for cases where data is retrieved from, respectively, cache or disk. The authors have pointed out that these two models were significantly different, and consequently designing a controller to satisfy both conditions was difficult. They further mentioned that a single fixed model cannot handle dynamically changing concurrent workloads, different operating conditions or application configuration changes. Similar arguments are made in Lu et al. (2002) for the case of a cache server system. Lu et al. (2006) implemented a control theoretic approach to achieve absolute and relative delay guarantees for Apache web servers. In their experiments, three different models for three different workload conditions were constructed. However, the control system was designed using a model generated by a one workload condition. Such design decisions may lead to performance issues when the other two workload conditions exist. Similarly, the authors of Zhu et al. (2006) and Wang et al. (2005) discussed the bimodal (under-loaded and overloaded) characteristic in time-varying workloads of data centers, and concluded an intelligent switching control approach is needed for these conditions.

We refer to the nature of a software system needing to operate across multiple regions and therefore needing multiple models to characterize the system behavior as the multi-region or multi-model characteristic of the system. To capture the multi-model characteristic of a software system, the possible solutions could be: (i) design multiple static models that can capture and cope with the system behavior in different operating regions; (ii) develop models and algorithms that can adapt and learn at runtime; or (iii) some combination of (i) and (ii). After capturing the behavior, multiple controllers have to be designed and integrated to the control loop or system, including the ability to detect the change of the operating regions and to switch between the appropriate controllers at runtime. These solutions demand control (decision making) systems that can dynamically reconfigure themselves and select the suitable model and controller. Furthermore, such runtime reconfigurations should achieve the desired performance objectives while maintaining the stability of the system (Salehie and Tahvildari, 2009, Brun et al., 2009). In this paper we call such control systems self-managing control systems. A self-managing control system provides a high-level of adaptive capability, because it reconfigures the structure of the control system (or the decision making component) at runtime, according to the changes in operating regions.

In this paper, we propose to use a control engineering approach, called Multi-Model Switching and Tuning (MMST) adaptive control, as the basis to implement and integrate self-managing control systems into software systems. This approach uses multiple system models and controllers, and automates the required parameter tuning and model/controller switching tasks to achieve the desired performance management in a way that takes account of the different operating regions of a software system. The designed self-managing control system autonomously detects the change of operating regions and then selects the most suitable controller to provide control decisions in that particular operating region at runtime. With the use of multiple system models and controllers, it also offers greater design flexibility than single model approaches. To assess its effectiveness in achieving performance management for software systems, we have conducted a range of experiments by applying two MMST adaptive control schemes to a representative software system with the multi-region characteristic. The performance of the two suitable MMST adaptive control schemes are compared to that of other single model control regimes under different operating conditions. The impact of configuration parameters for the MMST self-managing control schemes on the overall performance of the system is also investigated, providing guidance to the control system design. A reference model and class library based on the reference model is also introduced to support the implementations of MMST adaptive control schemes, which can be used by the software engineers in developing self-managing control systems.

The remainder of the paper is organized as follows. Section 2 provides a background of control engineering techniques and the related work. In Section 3, we describe a motivating scenario and prototype software system which will be used in the rest of this paper. After giving an overview of MMST adaptive control schemes in Section 4, we introduce a reference model and tool in Section 5 to aid the implementation of such MMST adaptive control schemes for a software system. Using the tool support, the design and implementation details of two self-managing control systems are covered in Section 6. The experiment results are presented in Section 7. Finally, a discussion, threats to validity and conclusions with an outline of challenges and future work are provided in 8 Discussion, 9 Threats to validity and challenges, 10 Conclusions, respectively.

Section snippets

Background and related work

In Section 2.1, we provide a background to different control system configurations that have been proposed in existing literature for performance management in software systems. Given that background Section 2.2 presents the related work.

Motivating scenario

To highlight the existence of multiple operating regions in software systems, let us consider a travel (flight) reservation system or server that serves different travel agents (see Fig. 2). The travel agent websites provide functionalities to their clients to check or book flights online. The travel reservation system implements the business logic needed by agents depending on the functional and QoS requirements. The travel reservation system also depends on a 3rd party supplier who holds the

Adaptive control with Multi-Model Switching and Tuning

In this section, we provide an overview of the adaptive control approach with Multi-Model Switching and Tuning (MMST) (Narendra and Balakrishnan, 1997) which supports various multi-model schemes. The MMST adaptive control was proposed by Narendra and Balakrishnan (1993) to improve the transient response of adaptive control systems in the presence of model uncertainties. It is a concept inspired by biological systems (Narendra and Driollet, 2000). Biological systems have the ability to select an

Reference model

In this section a reference model to design and develop MMST based self-managing control systems for software systems is provided. Fig. 5 shows the reference model at a high-level of abstraction. This reference model can be used in a number of ways to design self-managing control systems. Below, we first discuss the structure of the reference model. Then we show how this reference model can be used to integrate different control algorithms and switching schemes, before presenting the

Implementation of MMST adaptive control schemes

To quantify and assess the applicability and effectiveness of applying the MMST adaptive control schemes (types 2 and 4 in particular) to software systems, in this section we provide a systematic process to design, implement and integrate the MMST schemes using the prototype system introduced in Section 3 as a specific case study system. In Section 6.1, the model identification process to capture the different operating regions will be presented. Utilizing the class library introduced in

Experimental results

In this section, the performance of MMST-T2 and MMST-T4 schemes as designed in 6.2 Implementation of MMST-T2 scheme, 6.3 Implementation of MMST-T4 scheme is investigated in different operating conditions and business scenarios using the prototype travel reservation system with relative guarantee control. In order to do a comparative performance analysis, we also use two fixed gain controllers (controller-A and controller-B) and a self-tuning PI regulator (with the same settings as the MMST-T4

Discussion

The experiment results under the different operating conditions in Section 7 indicate that the MMST schemes (in particular, the type 2 scheme) are a promising approach to capture the behavior and to provide control in multiple operating regions of the software system by implementing self-managing (reconfiguring) control at runtime (for the case of relative guarantee over different client classes). Depending on the operating conditions, a suitable controller can be selected from the available

Threats to validity and challenges

The experimental evaluations presented in this paper are limited to a single case study system, even though it is based on a typical real world scenario deployed in a physical environment. In addition, the management problem investigated is based on the relative performance management scheme with two client classes. Therefore, the results presented in this paper are only strictly valid under these settings. In order to further confirm and generalize the validity, further experiments for

Conclusions

It is vital to incorporate self-adaptive capabilities into software systems in order to achieve performance objectives in dynamic and unpredictable environments. Control engineering techniques have been explored in the past few years to automate performance management tasks in software systems. However, existing control techniques based on single fixed or adaptive models are unable to provide satisfactory control in multiple operating regions of a software system under changing operating

Tharindu Patikirikorala is with Faculty of Information and Communication technology, Swinburne University of technologies, Australia (email: [email protected]).

References (50)

  • X. Zhu et al.

    Utility-driven workload management using nested control design

  • K.J. Åstrom et al.

    Adaptive Control

    (1995)
  • T.F. Abdelzaher et al.

    Web server QoS management by adaptive content delivery

  • Y. Brun et al.

    Engineering self-adaptive systems through feedback loops

    Software Engineering for Self-Adaptive Systems

    (2009)
  • Cheng, S.-W., 2008. Rainbow: cost-effective software architecture-based self-adaptation. Ph.D. Thesis, Carnegie Mellon...
  • Y. Diao et al.

    Managing web server performance with autotune agents

    IBM Systems Journal

    (2003)
  • Y. Diao et al.

    Using mimo linear control for load balancing in computing systems

  • R.C. Dorf et al.

    Modern Control Systems

    (2000)
  • X. Dutreilh et al.

    From data center resource allocation to control theory and back

  • D. Filev et al.

    Intelligent adaptive control using multiple models

  • N. Gandhi et al.

    Mimo control of an apache web server: modeling and controller design

  • D. Garlan et al.

    Rainbow: Architecture-based self-adaptation with reusable infrastructure

    Computer

    (2004)
  • Goel, A., Steere, D., Pu, C., Walpole, J., 1998. Swift: a feedback control and dynamic reconfiguration toolkit. Tech....
  • G. Gregorcic et al.

    A comparison of multiple model and pole-placement self-tuning for the control of highly nonlinear processes

  • J.L. Hellerstein et al.

    Feedback Control of Computing Systems

    (2004)
  • J.L. Hellerstein

    Self-managing systems: A control theory foundation

  • N. Kandasamy et al.

    Self-optimization in computer systems via on-line control: Application to power management

  • A. Karimi et al.

    Robust adaptive control of a flexible transmission system using multiple models

    IEEE Transactions on Control Systems Technology

    (2000)
  • M. Karlsson et al.

    An adaptive optimal controller for non-intrusive performance differentiation in computing services

  • M. Karlsson et al.

    Triage: Performance differentiation for storage systems using adaptive control

    Transactions on Storage

    (2005)
  • M.M. Kokar et al.

    Control theory-based foundations of self-controlling software

    IEEE Intelligent Systems

    (1999)
  • D. Kusic et al.

    Risk-aware limited lookahead control for dynamic resource provisioning in enterprise computing systems

  • L. Lennart

    System identification: theory for the user

    (1997)
  • X. Liu et al.

    Adaptive entitlement control of resource containers on shared servers

  • C. Lu et al.

    A feedback control approach for guaranteeing relative delays in web servers

  • Cited by (0)

    Tharindu Patikirikorala is with Faculty of Information and Communication technology, Swinburne University of technologies, Australia (email: [email protected]).

    Alan Colman is with Faculty of Information and Communication technology, Swinburne University of technologies, Australia (email: [email protected]).

    Jun Han is with Faculty of Information and Communication technology, Swinburne University of technologies, Australia (email: [email protected]).

    Liuping Wang is with Faculty of Electrical and Computer Engineering, Royal Melbourne Institute of Technology, Australia (email: [email protected]).

    View full text