Evaluating Timeliness and Accuracy Trade-offs of Supervised Machine Learning for Adapting Enterprise DRE Systems in Dynamic Environments *

Several adaptation approaches have been devised to ensure end-to-end quality-of-service (QoS) for enterprise distributed systems in dynamic operating environments. Not all approaches are applicable, however, for the stringent accuracy, timeliness, and development complexity requirements of distributed real-time and embedded (DRE) systems. This paper empirically evaluates constant-time supervised machine learning techniques, such as artificial neural networks (ANNs) and support vector machines (SVMs), and presents a composite metric to support quantitative evaluation of accuracy and timeliness for these adaptation approaches.


Introduction
Emerging trends and challenges.Enterprise distributed real-time and embedded (DRE) systems manage resources and data that are vital to organizations or projects.Examples include shipboard computing environments, air traffic management systems, and recovery operations in the aftermath of regional or national disasters.These systems often adjust their operations based on their external environment.For example, search and rescue missions as part of disaster recovery operations can adjust the image resolution used to detect and track survivors depending on the resources available (e.g., computing power, network bandwidth) [1].
Many enterprise DRE systems autonomically monitor their environment and modify their modes as the environment changes since manual adjustment is too slow and error prone.For example, a shift in network reliability can prompt quality-of-service (QoS)-enabled middleware, such as the OMG Data Distribution Service (DDS) [2], to change mechanisms (such as the transport protocol used to deliver data) since some mechanisms provide better reliability than others in some environments.Likewise, cloud computing applications, where elastically allocated resources (e.g., CPU speeds and memory) cannot be characterized accurately a priori, may need to adjust to available resources (such as compression algorithms optimized for given CPU power and memory) at system startup.The mission(s) of the system could be jeopardized if these adjustments take too long.
One way to adapt enterprise DRE systems autonomically involves the use of policy-based approaches [3] that externalize and codify logic to determine the behavior of managed systems.Policy-based approaches provide deterministic response times to perform appropriate adjustments given changes in the environment.The complexity of developing and maintaining policybased approaches for enterprise DRE systems can be unacceptably high and compromise trustworthiness, however, since developers must determine applicable policies for different environmental properties.Moreover, developers must manage the interaction of policies to provide needed adjustments.
Machine learning techniques [4,5] support algorithms that allow systems to adjust behavior based on empirical data (e.g., inputs from the environment).These techniques can be used to support autonomic adaptation by learning appropriate adjustments to various operating environments.Unlike policy-based approaches, however, machine learning techniques can automatically recognize complex sets of environment properties and make appropriate decisions accordingly.
Conventional machine learning techniques, such as decision trees and reinforcement learning, address autonomic adaptation for non-DRE systems [6].These techniques are not well-suited for enterprise DRE systems, however, since they do not provide data-independent bounded times when determining adjustments [7].Reinforcement learning techniques [8] explore the solution space until an appropriate solution is found, regardless of the elapsed time.Decision tree techniques have time complexities dependent upon the specific data and cannot be determined a priori.Moreover, decision trees may contain branches that are much longer than others, making the determination of appropriate adaptations unpredictable, which is undesirable in DRE systems.
Supervised machine learning techniques use training data to guide learning [9].These techniques can provide constant time complexity along with perfect accuracy in determining appropriate adaptations for environments on which they have been trained (i.e., known a priori).Techniques that trade off generality for specificity with perfect accuracy (i.e., are specialized for the environments they have seen and on which they have been trained) are called -overfitted‖ [10], which makes the accuracy equal to policy-based approaches (i.e., 100% accurate).Moreover, several techniques with constant time complexity that are not overfitted provide high accuracy for determining adaptations for environments unknown until runtime.Some techniques provide lower response times with lower accuracy, whereas others provide higher accuracy and response times.It is hard to manage (1) overfitted and non-overfitted techniques, (2) the accuracy and response times for these constant time techniques, and (3) the trade-offs between them.Developers must empirically evaluate the techniques, combine the results of accuracy and response time manually, and determine the most appropriate technique.
Solution approach → Integrated supervised machine learning techniques and composite metrics to guide trade-offs of accuracy and response times.This paper describes our timely-integrated machine learning (TIML) approach to integrating the following techniques: (1) overfitted supervised machine learning to respond perfectly to environments known a priori, (2) non-overfitted techniques to respond to environments unknown until runtime with high accuracy and constant response times, and (3) a composite metric to evaluate the accuracy and response time of different techniques quantitatively.TIML supports low-latency, constanttime complexity for determining adaptations to operating environments, 100% accuracy for environments known a priori, and high accuracy for environments unknown until runtime.We evaluate techniques for managing both response times and accuracy.
Our prior work [11,12,13] presented an architecture for autonomic adaptation and evaluated machine learning techniques without pinpointing the fastest response times.The work presented in this paper adds new experimental data and analysis, including fastest response times, as well as providing a new composite metric to evaluate accuracy and response time.We (1) overfit an artificial neural network (ANN) [14] (which is a technique modeled on interactions of neurons in the human brain) to retain as much information about specific environment configurations and adjustments as possible (e.g., greatly increasing the number of connections between input environment characteristics and output adjustments used in an ANN), (2) integrate nonoverfitted ANNs and support vector machines (SVMs) [15] (which generate the boundaries between different groupings to maximize the differences between groupings and aid in classification) to provide low response times and high accuracy for environments unknown until runtime, and (3) evaluate the machine learning techniques using the AccuLate metric that quantitatively combines accuracy and latency.Our ADAptive Middleware And Network Transports (ADAMANT) framework integrates TIML with the DDS QoS-enabled middleware to ensure accurate, timely, and predictable adaptation to dynamic environments.
The remainder of this paper is structured as follows.Section 2 provides a motivating example to illustrate the need for evaluating both timeliness and accuracy of machine learning approaches.Section 3 outlines key challenges for enterprise DRE systems in dynamic environments.Section 4 presents our solution approach of integrating multiple machine learning techniques and using composite metrics for quantitative evaluation.Section 5 describes our experimental results.Section 6 compares our work with related research.Finally, Section 7 concludes with lessons learned.

Motivating Example -Search and Rescue (SAR) Operations for Disaster Recovery
To motivate the need for integrating machine learning techniques, this section describes the challenges associated with search and rescue (SAR) operations.SAR operations are part of disaster recovery enterprise DRE systems which manage relief efforts in the aftermath of a disaster, such as a hurricane or earthquake.SAR operations help locate and extract survivors in a large metropolitan area after a regional catastrophe.SAR operations use unmanned aerial vehicles (UAVs), existing operational monitoring infrastructure (e.g., building or traffic light mounted cameras intended for security or traffic monitoring), and (temporary) datacenters to receive, process, and transmit event stream data from sensors and monitors to emergency vehicles that can be dispatched to areas where survivors are identified.Fig. 1 shows an example SAR scenario where infrared scans along with GPS coordinates are provided by UAVs while video feeds are provided by existing infrastructure cameras.These infrared scans and video feeds are then sent to a datacenter, where they are processed by fusion applications to detect survivors.Once a survivor is detected the application can develop a three dimensional view and highly accurate position information so that rescue operations can commence.

Key Challenges of Enterprise DRE Systems
This section summarizes key challenges that arise when developing autonomic enterprise DRE systems, such as the datacenter in the SAR motivating example in Sec. 2.

Challenge 1: Timely Adaptation in Dynamic Environments
Due to the dynamic environment inherent in enterprise DRE systems, application operations (such as image compression to reduce network traffic or disseminating data with timeliness and reliability properties) must adjust in a bounded-ideally constant time-manner as the environment changes.Operations that cannot adjust quickly and in a bounded amount of time will fail to perform adequately when resources change.For example, if resources are lost or withdrawn-or demand for information increases-operations must be configured to accommodate these changes with appropriate responsiveness to maintain a minimum level of service.If resources increase or demand decreases, operations should adjust as quickly as possible to provide higher fidelity or more expansive coverage.Manual modification is often too slow and error prone to maintain QoS.

Challenge 2: Accurate Adaptation to Dynamic Environments
Application operations in enterprise DRE systems must accurately adjust to changes in the environment.As changes in enterprise DRE systems occur (e.g., increases in networking capability or requests for data from new senders and receivers), the system should take advantage of additional resources or provide access to additional data producers and consumers while maintaining QoS.For a given environment configuration, the enterprise DRE system must accurately implement adjustments to attain an appropriate response and fully leverage existing resources.

Challenge 3: Flexibility in Trading Off Accuracy and Timeliness
Application operations in enterprise DRE systems must be able to trade-off adaptation accuracy with timeliness.
The situation may demand that finding a less accurate adaptation in time is better than finding an ideal adaptation too late [16].For example, selecting an adaptation that responds more quickly but has a lower probability of accuracy may be needed when response time is crucial (e.g., failure of critical infrastructure is imminent or groups of injured survivors must be detected quickly).If perfect detection of survivors is performed too late, it may not be possible to rescue the survivors.

Solution Approach -Integrating Machine Learning Techniques and Composite Metrics
Timely integrated machine learning (TIML) integrates multiple machine learning techniques to provide both (1) perfect accuracy and low response latency in determining appropriate adjustments, such as adjustments to transport protocols to support QoS in dynamic environments, for environments known a priori and (2) high accuracy for environments unknown until runtime.The AccuLate composite metric provides quantitative guidance for balancing accuracy and response time latency.This approach enables enterprise DRE systems to adjust to their environments autonomically and evaluate accuracy and response latency quantitatively.Moreover, we leverage techniques that provide the constant time complexity assurance needed for enterprise DRE systems.TIML overfits ANNs to retain a high degree of information about specific environment configurations and adjustments (e.g., increasing the number of hidden nodes used in an ANN).Hidden nodes are the computational components that provide connections between the relevant properties of the operating environment (e.g., CPU speed, network reliability) with the adjustments needed for those environments.As the ANN learns, it strengthens or weakens the connections between inputs, hidden nodes, and outputs to provide appropriate adjustments.Increasing the number of hidden nodes increases the level of detail that the ANN maintains.Moreover, TIML utilizes SVMs configured with different kernels (i.e., approaches to generating additional features from the environment configurations [17]) to increase accuracy over ANNs for environments unknown until runtime.Our approach resolves the challenges presented in Sec. 3 as described below.
 Machine learning techniques that use a static number of equations for learning address Challenge 1 in Sec.3.1 by providing predictable time complexities for determining appropriate adjustments.In particular, we apply overfitted ANNs for environments known a priori and multiple machine learning techniques for environments unknown until runtime to support enterprise DRE systems by incorporating the appropriate QoS-enabled middleware and transport protocol adjustments based on accuracy and timeliness concerns.When machine learning techniques are used in an enterprise DRE system, the time needed to make an appropriate adjustment is bounded by a constant number of equations.

 Integrating machine learning techniques address
Challenge 2 in Sec.3.2 by increasing the accuracy for environments known a priori and increasing the accuracy for environments unknown until runtime.Our approach increases the accuracy of determining appropriate adjustments by using an overfitted ANN for environments known a priori and integrated machine learning techniques that provide increased accuracy as compared to overfitted ANNs.Specifically, overfitting ANNs provides accuracy equal to policy-based approaches for environments known a priori, while non-overfitted techniques increase accuracy for environments unknown until runtime.

 Incorporating multiple machine learning techniques
and evaluating accuracy and timeliness simultaneously addresses Challenge 3 in Sec.3.3 by sup-porting multiple techniques with different levels of accuracy and response times for environments unknown until runtime and providing a composite metric to evaluate the trade-offs of these different techniques quantitatively.TIML supports ANNs and SVMs with various configurations.Integrating these ANNs and SVMs provides flexibility to support timeliness and accuracy for systems that need to balance the two concerns.Moreover, the AccuLate composite metric described in Sec.5.3 allows quantitative evaluation of these techniques.

Experimental Results
The section presents the results of experiments we conducted using ANNs and SVMs to determine timeliness, accuracy, and the balance between them to show the SAR datacenter leveraging ADAMANT in selecting an appropriate transport protocol configuration for a given operating environment.The experimental input data used to train the machine learning techniques include ADAMANT with multiple properties of the operating environment varied (e.g., CPU speed, network bandwidth, DDS implementation, percent data loss in the network), along with multiple properties of the application being varied (e.g., number of receivers, sending rate of the data), as would be expected with SAR operations.
We collected 394 inputs from previous experiments [18] where an input consists of data values that determine a particular operating environment (e.g., CPU speed, network bandwidth, number of data receivers, sending rate).We also provided the expected output to the ANNs and SVMs which is the transport protocol that provided the best QoS with respect to data reliability, average latency, and jitter (i.e., standard deviation of the latency of network packets).An example of one of the 394 inputs is the following: 3 data receivers, 1% network loss, 25Hz data sending rate, 3GHz CPU, 1Gb network, using the OpenSplice DDS implementation, and specifying reliability and average latency as the QoS properties of interest.Based on our experiments, the corresponding output would be the NAK-based multicast protocol with a 1 ms retransmission timeout.

Evaluating the Accuracy of ANNs and SVMs
We addressed the SAR accuracy requirement by first training the ANNs and SVMs on the 394 inputs mentioned above.We used the Fast Artificial Neural Network (FANN) library [19] as our ANN implementation due to its configurability, documentation, and opensource availability.FANN offers extensive configurability for the neural network including the number of hidden nodes connecting inputs with outputs.For SVMs, we used the libSVM library [20] due to its configurability, documentation, and open-source availability.
To determine the most accurate ANN and SVM we ran training experiments with the ANNs using different numbers of hidden nodes and SVMs with different kernels.Only one SVM, however, provided 100% accuracy for the environment configurations on which they had been trained (i.e., known a priori).For a given number of hidden nodes we trained the ANN 10 different times.The weights of the ANN determine how strong connections are between nodes.The weights are randomly initialized and these initial values have an effect on how well and how quickly the ANN learns.
Fig. 2 shows the accuracies for the ANN configured with 3, 4, 6, and 12 hidden nodes over 10 training runs.Fig. 2 also shows the effect of random initial weights on the accuracy of the ANN since the accuracy can vary across training runs.Accuracy was determined by querying the ANN with the data on which it was trained.Fig. 2: ANN Accuracy for Known Environments 100% accurate classification was generated at least once with all hidden node configurations except when using 3 hidden nodes.The ANN with 12 hidden nodes provided the best accuracy across all the training runs-100% accuracy all but 3 times out of 10 which would make it more likely to provide 100% accuracy for any single training run.However, once a single 100% accurate classification has been found it provides the maximum accuracy.Therefore we chose the ANN with 4 hidden nodes since it has the lowest response time as shown in Sec.5.2.
No ANNs configured with hidden nodes fewer than 4 and only a single SVM configuration provided 100% accurate classifications for the test data.We do not include the accuracy data for SVMs since the SVM response times are an order of magnitude greater than ANNs as shown in Sec.5.2.The training data values had to be scaled from -1 to 1 in order to achieve 100% accuracy for the configurations in Fig. 2. To evaluate accuracy with unknown environments we used 2-fold cross-validation, where 394 environment configurations are split into two mutually exclusive training and testing data sets [21].ANNs and SVMs are trained using training data and evaluated using testing data.The highest average accuracy for ANNs across the 10 training runs is produced with 12 hidden nodes (i.e., 76.09% average accuracy).The second highest average accuracy is produced with 6 hidden nodes (i.e., 72.84% average accuracy).SVMs produce higher accuracies, however, (i.e., 86.29% accuracy for SVMs using either the RBF or polynomial kernel).

Evaluating the Timeliness of ANNs and SVMs
As described in Challenge 2 in Sec.3.2, the datacenter for SAR operations requires timely configuration adjustments.This section provides timing information for ANNs and SVMs when queried for an optimal transport protocol.We used a 3 GHz CPU with 2GB of RAM running Fedora Core 6 with real-time extensions.Timeliness was determined by querying the ANNs and SVMs with all 394 inputs on which they were trained.A high resolution timestamp was taken before and after each call to the ANNs and SVMs.Fig. 5: ANN Average Response Times (μseconds) Fig. 5 shows the average response times for 10 separate experiments where for each experiment we query the ANN for each of the 394 inputs.The figures show that ANNs provide timely and consistent responses.As expected, the response times using more hidden nodes are slower than response times with fewer hidden nodes.The increase in latency is less than linear, however (e.g., response times using 12 hidden nodes are less than twice that using 6 hidden nodes).
Fig. 6 shows the response times for SVMs configured with different kernels and data scaling approaches.The SVM with the linear kernel tends to have the lowest response time with the polynomial and RBF kernels being the next most responsive respectively.Scaling the environment configuration input data and the transport protocol output response has an effect on the response times as well since this scaling affects the specific kernels that are created for the data.

Evaluating the Trade-offs of Accuracy and Timeliness for ANNs and SVMs
Deciding which machine learning technique to use for environments known a priori is straightforward.The ANN configured with 4 hidden nodes provides a configuration with 100% accuracy (shown in Fig. 2) and the lowest latency with a 100% accurate configuration (shown in Fig. 5).It is more challenging, however, to decide which technique to use for environments unknown until runtime.ANNs generally provide a lower response time, while SVMs provide higher accuracy.

Fig. 7: AccuLate Formula
We created the AccuLate composite metric to provide quantitative evaluation of machine learning techniques when considering both accuracy and response latency.As shown in Fig. 7, the AccuLate metric multiplies the inaccuracy percentage of a technique by its average latency.The number of total samples minus the number of correct classifications yields the number of inaccurate classifications.This result is divided by the number of total samples to produce inaccuracy as a fraction.We multiply the inaccuracy fraction by 100 to get the inaccuracy percentage.This multiplication by 100 gives the inaccuracy equal weight with the latency when multiplying the two values (i.e., the inaccuracy values range from 0 to 100 while the latency values for our current timing experiments range from single digits to double digits of microseconds).
We then add one to the inaccuracy percentage to account for perfect accuracy where the inaccuracy value would otherwise be zero making the entire AccuLate value zero.Adding one to the inaccuracy percentage allows AccuLate to produce a useful quantitative value for comparing machine learning techniques even when the techniques are 100% accurate.The utility of this adjustment is shown for some machine learning techniques (e.g., overfitted ANNs) when they are queried against the data on which they have been trained (i.e., known environments).
We use inaccuracy rather than accuracy as a factor in the AccuLate formula so that a technique that has both desirable qualities of high accuracy/low inaccuracy and low latency will produce a lower AccuLate value than a technique that has either (1) the same high accuracy and higher latency or (2) lower accuracy and the same latency.The inaccuracy percentage is based on 2fold cross-validation as outlined in Sec.5.1 which gives guidance as to how a machine learning technique will perform given an operating environment configuration on which it has never been trained.The AccuLate formula can be easily modified to use units of measurement other than microseconds for latency.We use microseconds since the techniques we evaluated all responded within 10s of microseconds.Fig. 8 shows the AccuLate values for the various machine learning techniques we evaluated for operating environments known a priori when 100% accuracy was achieved The ANN configured with 4 hidden nodes produces the best AccuLate value since it provides 100% accuracy (i.e., for 5 different classification runs) and the lowest overall latency.These values are equal to the values shown in Fig. 5 when 100% accuracy is achieved which highlights AccuLate's utility in comparing techniques when accuracy is equal.
Fig. 9 shows the AccuLate values for ANNs when operating environments were unknown until runtime.This figure shows that when accuracy and latency are given roughly equal weight (i.e., same order of magni-tude for values), the ANN with more hidden nodes provides a better balance of both accuracy and low latency for deciding an appropriate transport protocol for a given operating environment.The ANN with 12 hidden nodes consistently provides the best (i.e., lowest) Accu-Late values while the ANN with 3 hidden nodes provides the worst (i.e., highest) values.In particular, when no scaling of the data is done the SVM with the linear kernel produces the worst (i.e., highest) AccuLate values.Scaling the operating environment data to be between -1 and 1 produces the best AccuLate values for all the SVMs including the worst performing SVM using this scaling (i.e., with the RBF kernel) which produces results better than any other kernel using a different scaling approach.
AccuLate values can also be useful in evaluating machine learning techniques within a latency threshold.These values can aid real-time systems with deadlines where several techniques may exist that fulfill the timing requirements (i.e., are within the deadline).Based on response times shown in Fig. 5 and Fig. 6, if the average response time must be below 2.5 microseconds then the relevant techniques are ANNs configured with 3, 4, and 6 hidden nodes.For unknown environments, Fig. 9 shows that the ANN with 6 hidden nodes provides the best (i.e., lowest) AccuLate values relative to the required deadline with an average AccuLate value of 56.22 (as compared to values of 69.26 and 119.22 for ANNs with 4 and 3 hidden nodes respectively).

Related Work
In this section we present research work related to our work on the timeliness and accuracy of supervised machine learning for adaptation.

Integration of Computational Intelligence Techniques
De Castro et al. [22] have conducted research on the diagnosis of Alzheinmer's disease using a hybrid model of influence diagrams and multicriteria decision making methods.An influence diagram is a generalization of a Bayesian network.It provides a representation that is both easily understandable between decision-maker and analyst and mathematically consistent [23].Multicriteria methods address complex decision making problems where several people and several different criteria are involved [24].This work combines influence diagrams and multicriteria methods to aid in the determination of which questions are the most relevant in decision making.This work is done within the context of diagnosis of Alzheinmer's disease.Combining both influence diagrams and multicriteria methods provides a more robust and complete solution for decision support when uncertainty is a factor.However, this work does not focus on the timeliness concerns for DRE systems (as outlined in Challenge 1) as is provided by TIML and the AccuLate composite metric.
Huan et al. [25] propose a dynamic model for integrating multiple machine learning techniques.The goal of the model is to incorporate several learning techniques dynamically as needed to produce the best overall machine learning for the requirements of a given environment (e.g., the need for fast responses, the need for accurate responses).This work differs from our approach in TIML in that TIML allows concerns of both timeliness and accuracy to be considered concurrently (as presented in Challenge 3) and specifically addresses the concern of bounded-and ideally constanttimeliness needed for DRE systems (as outlined in Chal-lenge 1).The AccuLate composite metric is used to provide a quantitative evaluation in this regard.
Fan et al. [26] have developed a machine learning model that integrates the machine learning techniques of Bayesian Clustering by Dynamics (BCD) [27] and Support Vector Machines (SVM).The model is used to forecast short-term electricity prices.The BCD classifier partitions the space of the input training data into multiple subsets without a priori knowledge of the criteria used for classification.The SVMs, using techniques for data classification and regression based on advances in statistical learning theory [28], are used for prediction in each subset.
This work serializes the use of BCD and SVMs (i.e., the output of the BCD is used as the input to the SVMs) while not being concerned with timeliness.In contrast, our work in TIML selects the best machine learning technique based on the quantitative values produced using composite metrics.TIML is also focused on the timeliness concerns of DRE systems (as presented in Challenge 1).

Machine learning in support of autonomic adaptation
The Dynamic Control of Behavior based on Learning (DCBL) middleware developed by Vienne and Sourrouille [29] incorporates reinforcement machine learning in support of autonomic control for QoS management.In addition to reducing the overall system knowledge required by the system developers, reinforcement machine learning also allows DCBL to handle unexpected changes.System developers are required to provide an XML description of the system.DCBL then uses this description together with an internal representation of the managed system to dynamically select the appropriate QoS.However, since reinforcement learning is unbounded in its time complexity, DCBL's use of reinforcement learning does not address Challenge 1 (timely adaptation).The focus of DCBL is also on single computers rather than addressing scalable DRE systems.Moreover, DCBL requires developers to specify in an XML file the selection of operating modes given a QoS level along with execution paths, which leaves managing interacting QoS to developers.Conversely, TIML automates the management of this complexity via machine learning techniques.
In work on Multicast Mapping (MCM), Tock et al. [30] utilize machine learning for data dissemination to manage the scarce resources of multicast groups.MCM hierarchically clusters data flows so that multiple topics are mapped onto a single session and multiple sessions are mapped onto a single reliable multicast group.MCM's approach manages the availability of multicast addresses in large-scale systems where these kinds of resources can be scarce.MCM leverages machine learning to adapt as user interest and message rate change during different times of the day.MCM is designed only to address the scarce resource of IP multicast addresses in large-scale systems, however, rather than focusing on timeliness (as outlined in Challenge 1) and managing the trade-off between accuracy and timeliness (as outlined in Challenge 3) which are addressed with TIML.

Classification Techniques for Knowledge Discovery
Wu et al. [] present work in diagnosing students with learning disabilities (LDs) using a knowledge discovery and classification tool.Previous work had been done in this area using ANNs and SVMs [32,33].However, assessors of LD students prefer knowledge discovery for verification rather than relying on the more opaque classifications of ANNs and SVMs.In this regard, rough set theory (RST) is a promising option since it not only performs as a classifier but can produce meaningful explanations for the classifications.Experimental results from this work show that RST is a competitive approach for knowledge discovery and produces greater prediction accuracy than other rulebased approaches (e.g., decision tree, ripper algorithms).Preprocessing has been added (e.g., clustering, discretization, reduct calculation) to improve the quality of the rules generated by the RST.However, this work is not concerned with classification timeliness (as outlined in Challenge 1) nor the trade-offs between timeliness and accuracy (as outlined in Challenge 3) in contrast to our work on TIML and the AccuLate composite metric.

Concluding Remarks
This paper illustrated the need to evaluate both the timeliness and accuracy of machine learning approaches for adaptive enterprise DRE systems.We presented the AccuLate composite metric to quantitatively evaluate these QoS concerns together.Moreover, the results of the experiments presented in this paper show how integrating ANNs and SVMs help address the timeliness, accuracy, and the trade-offs between them for these kinds of systems.Below we describe some lessons learned from our work on TIML:  ANNs provide perfect accuracy and low latency for guidance in operating environments known a priori.Our experiments showed that ANNs accurately determined which protocol supported the desired QoS for operating environments known a priori.Several different configurations of ANNs were able to provide perfect accuracy.We chose the ANN with the least number of hidden nodes that still provided 100% accuracy since this ANN also provided the lowest response latency.
 SVMs provide higher accuracy than ANNs for operating environments unknown until runtime at a cost of higher response latency.Our experiments showed that SVMs increased accuracy in determining which protocol supported the desired QoS for operating environments unknown until runtime.SVMs produced a 13% increase in accuracy over ANNs when comparing the most accurate SVM with the most accurate ANN (i.e., 86.29/76.09-1 = 0.13).ANNs produced a 91% decrease in response time over SVMs, however, when comparing the most responsive SVM to the most responsive ANN (i.e., 1 -1.84/21.23 = 0.91).
 Integrating ANNs and SVMs can leverage the strength of both approaches with the AccuLate providing quantitative comparisons.When ANNs and SVMs are integrated together in constant-time, DRE systems in dynamic environments can leverage the low response time and accuracy of ANNs for operating environments known a priori and the accuracy of SVMs for environments unknown until runtime.When the timeliness constraints of the system preclude certain techniques, the AccuLate metric can be used to determine which available technique provides the best mix of accuracy and response latency.
 Scaling the environment configuration and transport protocol data affects accuracy.We were not able to produce an ANN with 100% accuracy for environments known a priori if the data were not scaled.Moreover, SVMs sometimes produced their most accurate results for environments known a priori when the data was not scaled while for environments unknown until runtime scaling the data produced the best accuracy.
Additional information, papers, and source code for the technologies and tests are available in open-source form at www.dre.vanderbilt.edu/~jhoffert/ADAMANT.

Fig. 3 :
Fig. 3: ANN Accuracy for Unknown Environments Fig. 3 and Fig. 4 present accuracy results for ANNs and SVMs respectively for operating environments unknown until runtime.The ANNs are randomly assigned initial weights for the connections between nodes which accounts for variations in accuracy across training runs.We average the accuracy results across all runs below.The accuracy for SVMs is dependent upon scaling of the input and output training data.The different scaling scenarios are presented in Fig. 4.To evaluate accuracy with unknown environments we used 2-fold cross-validation, where 394 environment configurations are split into two mutually exclusive training and testing data sets[21].ANNs and SVMs are trained using training data and evaluated using testing data.The highest average accuracy for ANNs across the 10 training runs is produced with 12 hidden nodes (i.e., 76.09% average accuracy).The second highest average accuracy is produced with 6 hidden nodes (i.e., 72.84% average accuracy).SVMs produce higher accuracies, however, (i.e., 86.29% accuracy for SVMs using either the RBF or polynomial kernel).

Fig. 10
Fig. 10 shows the AccuLate values for SVMs when operating environments were unknown until runtime.This figure shows that generally the SVM with the linear kernel produces the best (i.e., lowest) AccuLate value.The figure also highlights that scaling the data (i.e., the input operating environment and the output transport protocol) has an effect on the AccuLate values due to the corresponding change in accuracy.