Beyond the Hype: An Evaluation of Commercially Available Machine Learning–based Malware Detectors

1 INTRODUCTION

Attackers use malicious software, known as malware, to steal sensitive data, damage network infrastructure, and hold information for ransom. One of the top priorities for computer security tools is to detect malware and prevent or minimize its impact on both corporate and personal networks. Traditionally, signature-based methods have been used to detect files previously identified as malicious with near perfect precision but potentially miss newer malware samples. With the advent of self-modifying malware and the rapid increase in novel threats, signature-based methods are insufficient on their own. By generalizing patterns of known benign/malicious training examples, machine learning (ML) exhibits the capability to quickly and accurately classify novel file samples in many research studies [19]. Moreover, ML-based malware research has made the transition from the subject of myriad research efforts to a current mainstay of commercial-off-the-shelf (COTS) malware detectors. Yet few practical evaluations of COTS ML-based technologies have been conducted.

Turning from the academic literature to market reports from commercial companies can provide (for a fee) useful information, specifically, end-user feedback, itemization of all technologies in the antivirus/endpoint detection and response marketplace [17], and even statistics showing the efficacy of the detectors on malware tests [4, 40]. (See Related Works 2.0.2 for recent examples by AV-TEST and SE Labs.) Yet, these tests often report near perfect detection for many tools, diminishing trust in the capability of the test to sufficiently stress test the tools or to differentiate the tools. Further, as our results—under 60% recall of all tools tested—and other previous works [10] confirm, these near perfect evaluations promote a false narrative that modern (especially ML-driven) malware detectors provide just shy of perfect protection. Even when armed with many statistics on detection capabilities, balancing or weighing these heterogeneous measurements when reasoning about the efficacy of tools is difficult [20].

As a result, organizations have limited insight into whether or how COTS malware detection tools add value to their current defense solutions, short of trusting possibly skewed vendor-provided claims and market/consumer reports. The origin of this article is evidence of this fact, as the authors were tasked with evaluating four market-leading malware detection tools to assist in understanding their efficacy. This work describes a scientific experiment designed and conducted by the authors (professional researchers) to assist an organization in examining the benefits and tradeoffs of commercially available ML-based malware detection tools.

Guided by the organization’s requests, we established meetings with 22 sales and technical representatives of many popular vendors to learn the merits and approaches of their detection technologies. Based on the organization’s requirements, four prominent detectors were chosen for study: two host-based (one signature based and one ML based) and two network-based (both ML based, one static and one dynamic) detectors. Of particular interest in this study are detection capability across varying file types and categories, efficacy of the network-level detectors across varying protocols, and studying value provided by ML- vs. signature-based, host vs. network, and static vs. dynamic tools, with an eye toward “defense-in-depth” (i.e., strategically combining tools). Our overall goal is to provide the funding organization with methods for quantitatively reasoning about malware detectors and to exhibit results on these four tools that can provide insights into two questions:

(Q1) ML Generalization Hypothesis: We are interested in the fundamental promise of the three ML-based malware detectors, namely “Can ML-based COTS tools accurately classify both never-before-seen files (especially zero-day malware) and publicly available files?”

(Q2) Network-level Malware Detection Hypothesis: Network-level malware detectors seek to complement host detectors. They leverage the advantage of greater computational resources (available in a network appliance, generally a commodity server) without affecting users’ workstations. This means, in theory, they can afford more in-depth analysis without impacting usability, namely slowing processes on each workstation. Notably, the network-level detector must carve the file out of the packets to correctly process it, which is not necessary for host detectors. Of course, if the network detector cannot expand the detection capabilities, then it simply adds cost without enhancing defenses. In our experience with approximately 10 security operation centers (SOCs) [7, 32], all centers required endpoint malware detection and signature-based network intrusion detection, but none used network-level malware detection technologies. Thus, we investigate the question, “Is it worth purchasing, configuring, and maintaining a network-level malware detector, given in-place host-based detection?”

We were given licenses from four vendors for experimentation under the agreement that results could only be released with anonymity; we are unable to provide any details that might disclose the vendors or their intellectual property. The four COTS tools used in this experiment—two endpoint detection tools, one signature based and one claiming to be solely ML based (supervised learning with no signatures), and two network-level tools, both ML based with one claiming static analysis (does not execute files) and the other claiming dynamic analysis (executes files in a sandbox)—provide representatives of different available malware detection approaches. The first endpoint malware detector being solely signature based is considered a baseline to represent the status quo for modern commercial malware defenses. Both network detection tools perform file extraction from packets (whereas endpoint detectors scan files on each host), and both claim to use ML techniques to detect files. The first tool uses static analysis and an ensemble of ML- and signature-based detectors. The second tool performs dynamic analysis to feed an ML classifier. See Section 3.2. We note that two types of commercially available malware detectors are not represented in this study: those that leverage a cloud connection for interactive intelligence (which can be at the host and/or network level) and network-level detectors that are solely signature based.

To test these technologies, we configured a network at the National Cyber Range (NCR) [14] (see Section 3) to deliver 2,554 malware and 982 benignware samples of varying file types to hosts. Zero-days (specially crafted malware never seen in the wild), files to mimic advanced persistent threat (APT) actions, and polyglots (files functioning with multiple file types) were also included to test the capability of these tools to identify novel threats (see Figure 1).

Fig. 1.

View Figure

Statistical results (Section 4) from the experiment give enlightening insight into how well or how poorly commercially available malware detectors perform. Analysis of detection statistics by file type and delivery protocol illuminates rather alarming gaps. Comparison of the signature-based tools to the three ML tools, especially on zero-day files, allows us to quantify question (Q1), the ML generalization hypothesis. Furthermore, we provide an analysis of conviction latency (i.e., time to detect) per tool and per file type to illustrate the gains/losses of network tools and of using static versus dynamic analysis. Under the assumption that host-based malware detection is a requirement, we consider a defense-in-depth question, “Is it worthwhile to use multiple detectors together?,” by simulating complementary detectors using the logical OR of their alerts.

While these statistical results are enlightening, they present tradeoffs that are difficult to reconcile when choosing the “best” malware detection tools (e.g., how to balance give and take among detection rates, false alerts, detection delays, and anticipated attack costs). To aid the funding organization in using all the results to assess the tools, we design a novel instantiating of the general cost–benefit framework of Iannacone and Bridges [20] to quantifiably assess and compare the network-level tools (question Q(2)). This method simulates the cost to an enterprise using the technologies by integrating detection accuracy and time-to-detect statistics (all learned from experimental testing), along with estimates of attack damage costs, labor costs, and resource costs. We configure the cost model to estimate costs of the two network-level tools on the emulated network. Then in a separate evaluation, we estimate the additional costs for adding the network tools to complement each endpoint tool. Our configuration of the cost model, in particular in evaluating complementary detectors used in tandem, is novel. It gives new vantage points for understanding the tools’ efficacy use and provides a procedure to assist future acquisition decisions. See Section 5.

2 RELATED WORK

A large body of work proposes and tests ML algorithms for malware detection, and surveys have collected and organized these works. However, to our knowledge, few works seek to evaluate commercial, especially, ML-based malware detection, and none provides cost–benefit analysis informed by tool-specific detection statistics learned from experimental results for reasoning about these detectors. In addition, the most thorough existing evaluation of malware detection tools in the literature uses a corpus of 1,000 samples, compared with our evaluation of more than 3,500 samples. Finally, no previous work has the efficacy of pairing complementary detectors as suggested by the “defense-in-depth” idea. Here we address commercial evaluations, which are much larger than ours, but do not provide differentiation of tools—many tools get near perfect results—inducing skepticism of the efficacy of these evaluations.

3 EXPERIMENTAL DESIGN

To execute the experiments, we leveraged the NCR, an air-gapped research test bed equipped with state-of-the-art network and user emulation capabilities to facilitate the experiment [14]. A variety of services were emulated, including email, website servers, and domain controllers, and they were used by emulated users both internal to the network and in the emulated worldwide web (external). We provide a network diagram in the Appendix (Figure A.1).

Traffic generation was provided on all client endpoint virtual machines (VMs) by the NCR Mantra software, a proprietary emulator built on Lariat [39]. All clients had broadly similar traffic generation profiles and were configured to automatically and randomly browse websites (both internal and external), send and reply to emails, generate Microsoft (MS) Office documents, transfer files to and from file shares and Sharepoint sites, and run a preset list of other benign executables. All clients were set to a fixed diurnal schedule of increased daytime activity between the hours of 0700 and 1,700 local time; activity outside of these hours was decreased by a factor of 10 (i.e., average inter-arrival time of traffic generation events was set to be 10 times as high).

System time was synchronized between all devices except the network static appliance via NTP to the central NCR testbed time source. Every node with an in-band connection synchronized to an NTP server in the emulated Internet environment, which in turn connected to the central time source via the testbed control network; everything with only an out-of-band connection (namely the network detection servers, a traffic collection server, and SOC workstations) connected directly via the control network to the same source. The network static appliance was supplied as a vendor-loaned server without administrative (root) access, which is required to change time settings; the offset of this appliance’s time vs. actual time was noted for compensation in after-action data analysis.

This environment enabled delivery of malware over the network to endpoints running the two host-level tools, so the network-level tools could (attempt to) reconstruct files from the packet stream and provide detection. In particular, emulated internal and external websites, mail servers, and hosts were used to deliver files (both benign and malicious) to internal clients through web download, email attachments, and direct connections, respectively.

The authors met with vendors of the technologies and configured the tools in advance of the NCR to ensure proper configuration. Configuration guides were used to duplicate the process at NCR, and, as a quality assurance check, detection results on a set of test files were confirmed to be identical before the experiment (with tools configured by the vendors) to after setup in the NCR. All technologies operated on-premises, without connections outside the internal portion of the emulated network. Although both host-level tools provide pre-execution conviction and quarantine, for the purpose of the experiment all tools were set to alert only (not block/prevent) to permit complementary (OR) analysis of multiple tools. All machines were time synchronized. For the network-level tools, a secure sockets layer (SSL) “break-and-inspect” decryption technology was not added to this network. Files were delivered on unencrypted protocols as the goal was to test the detectors’ capabilities to detect malware without dependency on the capabilities of the decryption technology. (In practice, under this configuration malware sent encrypted would almost certainly be opaque to these detectors. How such a decryption technology affects the network-level detector is out of scope for this work.)

For this experiment, we evaluated the endpoint detection on both Windows 7 and 10 for an endpoint (nonserver) host, although Linux hosts and servers were present. Windows 7 and 10 VMs were given 4 GB of RAM and 1 CPU core, along with a large disk drive for the installed operating system. RHEL6 VMs were given 1 GB of RAM and 1 CPU core, also with a large disk drive for storing the operating system and associated files. These values reflect the amount of minimum memory needed to create a VM for the specific operating system, with both the Windows VMs requiring at least 4 GB and RHEL requiring only 1 GB. The test bed was controlled via orchestration infrastructure leveraging virtualization software and custom wrapping software to coordinate with the VMs and manage the VM lifecycle.

The detection technologies were each exposed to 3,536 total files (2,554/982 = 72/28% malicious/benign) of varying categories, including 32- and 64-bit portable executables (PE32, PE64), MS Office, PDF, JAR, and three unusual categories: APT (described subsequently), zero-day PE (Windows PEs believed to not be available publicly), and polyglot (files that are valid as multiple file types, e.g., ZIP and PDF). Most malicious files were obtained through VirusShare.⁴ Our polyglot categories should also be never-before-seen files, and at least one of the valid file types has malicious functionality; specifically, we combined malicious Java JAR files with GIF and JPEG files. Notably, one can leverage open source software for polyglot creation ( https://github.com/corkami/mitra). The zero-day files were manufactured samples that never touched infrastructure with access to the Internet.

The goal of the APT class was to create artifacts that exemplified common APT evasion tactics. This included using signed vs. unsigned executables, cloning the PE headers of common benign executables, and advanced in-memory characteristics. Cobalt Strike⁵ provided a convenient way to test these features. Because Cobalt Strike is a known tool, we needed to minimize the risk that our artifacts would be discovered by generic Cobalt Strike signatures. To this end, we built a custom obfuscator for the samples in the APT class using the Cobalt Strike artifact kit. The artifact kit is effectively an API that allows users to change the way Cobalt Strike generates binary artifacts, allowing advanced users to more successfully evade AV technologies. As a quality assurance step, we built a named pipe bypass using the artifact kit and ensured that default samples (i.e., samples without any of the APT-like evasions) generated with this obfuscation were not detected by a fifth, commonly used malware detector. With this obfuscation in hand, the APT class was created by combining that obfuscation with the various APT-like evasions available in Cobalt Strike. For those APT samples that were correctly detected in our experiment, we cannot know if the correct detection was from a generic signature (that our obfuscator failed to evade) or if indeed the APT-like evasion tactic was correctly identified behaviorally.

3.1 File Samples

Our experiment used an imbalanced class, that is, a smaller number of benign files than malicious. See Figure 2. As the experiment involved instantiating an entire analysis environment and waiting up to 2 minutes for a detection decision on each file, time limitations forced our decision to limit the benign files. We biased the benign set to include a set of “tricky” benign samples generated and donated by researchers who developed the Hyperion tool [26, 36]. These benign files are mostly C/C++ compiled utilities and were often (incorrectly) classified as malicious in Hyperion research. The rest of the benign files are standard office type documents, JAR files, RPMs, DEBs, and EXEs/MSIs.

Fig. 2.

View Figure

Overall, inclusion of nonpublic malware and benignware allows testing of the ML generalizability hypothesis—that it can correctly identify never-before-seen files, while the variety of malware allows examination of recall across file types and categories.

4 STATISTICAL RESULTS

Here we provide the usual accuracy and conviction time statistics in aggregate, broken down by file type and, for the network-level tools, by delivery protocol. Furthermore, we present results for each detector side by side and in complementary combinations.

4.1 Individual Sensor Results

Table 1 itemizes the results of each malware analysis technology, with rows per file type except the bottom row, which gives overall detection statistics. Because a focus of this experiment is to identify the ways in which ML-based technologies add value, we consider the host signature-based sensor as a baseline. The host signature-based tool demonstrates perfect precision but poor recall, pulled down by its inability to detect never-before-seen malware (0% polyglots, 3.5% zero-days detected). The tool performs relatively well with respect to MS Office files and APT files and fairly well with respect to executable and PDF files. Good performance on executable, MS Office, and PDF file types is not surprising, because these are the most common packaging for malware and, therefore, have the broadest set of signatures. As expected, the host signature-based tool performed poorly on JAR files and zero-day samples, where there are fewer relevant signatures, or in the case of zero-day samples, no existing signatures.

Table 1.

Category	Host Signature			Host ML			Network Static			Network Dynam.
(#mal./#ben.)	Prec.	Recall	F1	Prec.	Recall	F1	Prec.	Recall	F1	Prec.	Recall	F1
APT (27/0)	1	0.667	0.8	1	0.63	0.773	1	0.296	0.457	1	0.556	0.714
JAR (325/200)	1	0.0892	0.164	0	0	0	1	0.249	0.399	1	0.209	0.346
Office (400/200)	1	0.782	0.878	0	0	0	1	0.873	0.932	1	0.735	0.847
PDF (400/200)	1	0.512	0.678	0	0	0	1	0.51	0.675	0.996	0.565	0.721
PE32 (400/200)	1	0.505	0.671	0.922	0.855	0.887	0.814	0.69	0.747	0.977	0.642	0.775
PE64 (400/182)	1	0.233	0.377	1	0.777	0.875	1	0.833	0.909	1	0.757	0.862
Polyglot (199/0)	0	0	0	0	0	0	0	0	0	0	0	0
Zero-Day (403/0)	1	0.0347	0.0671	1	0.486	0.654	1	0.395	0.566	1	0.553	0.712
Total (2554/982)	1	0.342	0.51	0.968	0.339	0.502	0.957	0.552	0.7	0.995	0.543	0.702

• Categories are disjoint in this table: all 403 zero-day files are PE32 files but are not counted in the PE32 row; similarly, all polyglot files are malicious JARs but are not counted in the JAR row. The Total row provides the statistics computed on all files.

View Table

Table 1. Individual Sensor Results Itemized

• Categories are disjoint in this table: all 403 zero-day files are PE32 files but are not counted in the PE32 row; similarly, all polyglot files are malicious JARs but are not counted in the JAR row. The Total row provides the statistics computed on all files.

The good performance on the APT category was a surprise. It is likely that the Cobalt Strike framework used for packaging the APT malware samples just happens to be well known to the signature set used in this host signature-based sensor. The host signature-based tool’s perfect precision simplifies this analysis, as recall becomes the primary metric of comparison. Performing worse in overall recall, the host ML tool has 0% detection for all non-PE files. The two network-level tools perform quite similarly, with good (\(\gt\)95%) precision and both with better (but still alarmingly poor) recall of 55%, giving F1 near 70%.

The network static tool achieves a recall of \(\hbox{$\scriptstyle \mathtt {\sim }$}\)20% greater than the host-based tools. As this tool uses a suite of ML- and signature-based subclassifiers, these results suggest that complementary signature and ML tools might work well. This increase in recall is shadowed by a higher quantity of FP alerts. Nearly identical overall statistics are achieved by the lone dynamic tool. Our results imply that network-level tools do increase recall substantially with comparable precision to the host ML tool and only slightly worse precision to the host static tool.

Overall, regarding Table 1, there is no clear “winner.” Precision values are high for all four tools, meaning all sensors performed with relatively few false alerts. Perhaps the biggest takeaway of these results is the utter failure of these tools to detect most malware. Recall of 33–55% is both disappointing and alarming, with roughly 37% of the malware used in the exercise undetected. Furthermore, recall varies wildly across file categories for each tool. Notably, most operations use a sole endpoint detector, and both of these state-of-the-art commercial detectors identified only a third of the malware used in this test. Polyglots are a glaring problem for these four tools, as no such file was detected. This suggests that the approach to detection is dependent on determination of file type.

Considering the zero-day category, we see that all three ML-based tools (host ML and both network tools) increase recall by more than a factor of 10. This quantifies our first hypothesis, at least for these tools, that ML, as incorporated into modern detection products, can help generalize to never-before-seen files. Simply put, in our results, the detection rates increase by approximately 10 times but at the cost of precision—specifically precision on PE32 files—which drops by approximately 10%. Considering only the zero-day PEs and the APT files, the network dynamic tool is (unsurprisingly) superior. Comparing this result with the amount of time needed to detect will put these figures in context.

4.1.1 Detection Latency.

In addition to accuracy, timely detection is imperative to effective defense against malware. Figure 3 shows each tool’s time from file delivery until alert both per file type with true positives (TPs) and FPs indicated by color and in aggregate across all files to visualize the overall alert time empirical distribution. Detection latency is calculated from the timestamp generated when the file transmission is completed until the timestamp of a given tool’s alert on the respective file and encompasses all processing, including any file carving or decompression required. Tools’ average conviction times rank as expected, but the results hold some unexpected takeaways. The network static tool has the fastest average conviction time of 15.3 s, perhaps due to the resources available for a network appliance (much more than an application on a host) and static technique (generally faster than dynamic detection). Interestingly, all its FPs occur in PE32 files that take maximum times to alert for this tool. (Without visibility into the detection mechanism of the tool, we cannot diagnose this phenomenon. We notified the vendors of this anomaly.) Both host-based tools take at most a couple minutes to alert, with the ML-based tool slower on average. Anomalously, the signature-based tool JAR files exhibit wide variance in time to alert. Recall that the host ML detector does not alert on many file categories and has many FPs on PE32 files, whereas the host signature-based tool has no FPs. The network dynamic tool has a much greater average conviction time of 1,039 s. This is understandable given the process of creating a sandbox environment and then executing the file in that environment to make a conviction decision.

Fig. 3.

View Figure

We dug into the detection latency of these network-level tools in an additional experiment. Both vendors provided updated network appliances, and we tested their time-to-detect capabilities in a separate, more realistic experiment. Each tool received a mirrored feed of a real network’s traffic blended with emulated network traffic, allowing more than 2,000 malware samples to be delivered safely to VMs but spread over a few hours. While seeing data rates of up to 7 GB/s, the tools achieved median detection times, from file delivery until alert, of under 1 s for the network static tool and 258 s = 4.3 m for the network dynamic tool. (Median is reported here because of outliers in the statistics that affected the mean.) The network-level tools had a significant bottleneck in the computational overhead of decompressing files. Introducing emulated network data had a positive effect on detection latency as it significantly decreased the rate at which compressed files were delivered to the tools despite dramatically increasing the amount of raw data to process—the tools received more network traffic but a decreased rate of compressed files to process. These results show that, indeed, there is a latency cost for using at least this dynamic analysis tool and that both the network tools’ latencies can increase by an order of magnitude if files cross the network too rapidly relative to the server’s computational resources.

4.1.2 Network-level Detectors Efficacy across Protocols.

The two network-level tools rely on extracting files from traffic and claimed accommodation of multiple protocols. As such, we ran a small test using multiple protocols. Our results revealed that visibility into those network streams was not as reliable as expected. All of the 3,536 file samples used in the analysis were delivered over the HTTP protocol. To test the network tools’ capabilities across protocols, 200 of the malware samples were also delivered over four additional protocols. To deliver files over multiple protocols, web downloads (HTTP), external file transfers (FTP), internal file transfer (SMB), attachments to emails (SMTP), and clear-text TCP via Netcat were used. Netcat was used to test a vendor’s claim that it could detect files sent in this manner. We present the counts of correctly detected samples for these 200 malware samples on the five total protocols in Table 2. The network static tool has an email client that was not used in this analysis, and so its performance on email was not determined. All network transmissions were unencrypted. Both network IDS tools require de-/un-encrypted traffic. These results show that the tools’ capabilities to detect varies greatly by protocol, and this does not include the complexity induced by the realistic assumption that these tools would sit downstream of a companion decrypter.

Table 2.

	Dynamic	Static
FTP	0/200	9/200
HTTP	145/200	156/200
SMB	0/200	38/200
SMTP	180/200	N/A
TCP	0/200	0/200

View Table

Table 2. Detection Counts for Network-level Detectors on 200 Malware Samples Each Sent on Five Protocols

5 COST MODEL EVALUATION

Although the aforementioned metrics provide a valuable statistical summary of these sensors’ performance, recent work by Iannacone and Bridges [20] provides a cost–benefit framework that models the real-world implications of using a particular tool, yielding a single, comparable cost metric to more easily reason out the tradeoffs implied in Section 4. For example, the network dynamic tool has an average detection time of 1,039.0 s, an FP rate of 0.71%, and a TP rate of 54.3%, whereas the network static tool detects on average in 15.3 s (faster) with an FP rate of 6.42% (worse) and TP rate of 55.21% (about the same). It is difficult to know which tool is best for a given SOC to adopt based on these basic statistics. It is even more difficult to choose the best pair of tools.

To assist with these determinations, we configured the cost model to evaluate these tools as standalone detectors (Section 5.2). This follows the original cost-model work [20] but makes one novel contribution to the basic model. Since the cost model as originally proposed depends on the accuracy statistics of the tool, we test with a relatively large number of both malicious and benign files to accurately estimate these inputs. Our change to the model is to compute average costs per benign-/malware and then scale the costs to respect two new model inputs, namely the expected number of files per year times the ratio of benign-/malware expected in the wild. This allows us to gain accuracy in both the detection statistics and the cost model. In Section 5.3 we create a new version of the model to compute savings of adding a network-based malware detector, assuming a host-based detector is in place. This new estimate is created to answer (Q2).

5.1 Cost Model Overview

This model estimates and sums labor and resource costs incurred for buying, configuring, and using the tool in addition to attack damage cost estimates. These costs are itemized by the following components. Refer to Table 5.

Table 5.

	Initial	Ongoing
Host Sig.	$2K	$8K/y
Host ML	$6.5K	$35K/y
Net. Static	$15K	$20K/y
Net. Dynam.	$23K	$16K/y

View Table

Table 5. Initial (One-time) Server or Appliance and Ongoing (Annual) Subscription Costs

\(C_I\), the one-time initial costs of purchases, setup, and installation. The initial configuration costs are based on our team’s experience with installing and configuring these devices at the NCR. All costs are slightly obfuscated to preserve anonymity of the tool vendors. This minimally impacts results but does not impact the rankings of the tools. All license, hardware, and labor costs assume a relatively small network (\(\hbox{$\scriptstyle \mathtt {\sim }$}\)1K IPs), comparable in size and complexity to the NCR test network described previously. Note that we ignore several smaller costs, such as power and cooling, because they are negligible.

\(C_B\), the ongoing (per month, year, etc.) costs of normal operation, including subscription-based licenses and labor needed for periodic updates, reconfiguration, and maintenance. Since we did not use the tools in a real setting for a long period of time, we simply cannot give per-tool estimates. We assumed 8 h/tool/month for all tools. This does not affect our comparison and was included for completeness.

\(C_{IR}\), the incident response (IR) cost (per true positive alert), representing labor costs for investigation and remediation of detected attacks. We estimate this at $280 = $70/h (fully burdened cost of SOC operator) \(\times\) 4 h.

\(C_T\), the alert triage costs (per alert), representing labor costs and storage costs for the alert’s data. We estimate this at $70/h \(\times\) 1 h. A SOC lead verified that these estimates of labor time for triage and (above for) IR were verified as reasonable. The triage resource cost of $0.05 was obtained by finding the lowest volume Splunk license based on the rate of raw data being processed ($1,800 for 1 GB/month, or 34 MB/day) and estimating what fraction of that volume would be consumed by each alert. For both detectors, we observed a peak of 1,000 alerts per day during our testing, and the average alert size was under 1 KB per alert, for a maximum of 1 MB/day. Because each detector is using less than 1/34th of the licensed volume for its 1,000 alerts per day, each alert represents a cost of (at most) $1,800/34/1,000 or about $0.05.

\(C_A\): While the above four costs are “defense costs,” \(C_A\) estimates the losses due to attack damages. Attack cost is a function of time modeling the average cost incurred from the moment of infection. We follow formulation of attack cost from the initial work [20] and use an “S” curve to model average attack cost over time. The intuition is that costs will begin at $0 and after some time (e.g., the malware unpacking itself) begin to grow quickly, then level out approaching a maximum cost. In reality, this might happen in a sequence of discrete jumps, but as discussed in the previous work, an “S” curve provides a reasonable model of the average cost for an attack. Specifically, the attack damage cost \(f(t)\) is defined as \(f(t) = M \exp (- (\alpha /t)^2 \ln {2}) = M 2^{-1/(t/\alpha)^2}\) where parameter M is the maximum cost (and horizontal asymptote), and parameter \(\alpha\) is the time when half the maximum cost is attained (\(f(\alpha) = M/2)\)). See depiction of \(f(t)\) in Figure 4 and input parameter choices in Table 6. The shape of the attack cost function, the parameters in Table 6, and the cost of labor hours used in this model were informed by discussions with a SOC lead who verified they are reasonable.

Fig. 4.

View Figure

Table 6.

Param. Value	Description
M = $2,000	Max. Attack Cost
\(\alpha\) = 15m = 900s	Half-Cost Time
N = 125K	# of Files
r = 1.16%	Malware Ratio

View Table

Table 6. Cost Model Parameters

To apply the model we first computed initial and ongoing costs, as explained previously, then compute each tool’s cost per sample as follows:

False Negative Costs:	If the tool does not alert on a malicious sample, then maximum attack cost is incurred (\(M = \lim _{t\rightarrow \infty } f(t)\)).
True Positive Costs:	If the tool correctly detects malware in t seconds, then the incurred cost is the sum of costs from triage resource and labor ($0.05 + $70), IR labor ($280), and attack damages \(f(t)\). Very late detection of malware can cost more than no detection, because, for large t, the attack cost is near maximal, and alert triage and remediation still occurs.
False Positive Costs:	If the tool incorrectly alerts on a benign sample, then triage costs are incurred ($0.05 + $70).
True Negative Costs:	If the tool (correctly) does not alert on benignware, then no extra costs are incurred.

Note that FN and TP costs are costs from malicious samples, whereas FP and TN costs are from benign samples.

We break up the sum of \(C_{IR} + C_T + C_A\) (incident response, triage, and attack costs) into two quantities—costs from false alerts plus from attacks (true alerts or no alert). To do so, we compute each tool’s average benignware cost \(a_b\) (from the remaining 992 benign samples) and average malware cost \(a_m\) (from the 2,554 malware samples). Finally, we set N, the number of unique files crossing the network detector (border traffic + internal-to-internal) for a 1,000 IP network, to be 125K files (125 per host). We also need an estimate of the ratio of malware to benignware in the wild. For this we leverage Li et al. [24], who upon investigating files on 10K IPs find 1.16% malware and 98.84% benignware. Thus, we can scale the average cost per benign or malicious file to 125K files per year, respecting the 98.84/1.16% benign-/malware ratio.

Putting it all together, the estimated cost for the first y years is as follows: (1) \(\begin{align} C &= C_I &&\text{initial resource and labor costs plus} \nonumber \nonumber\\ &+ y[ &&\text{years ($\times $ quantity)} \nonumber \nonumber\\ & \phantom{+ y[ + } C_B && \text{yearly ongoing costs, e.g., subscriptions and tweaking, plus} \\ & \phantom{+ y[ } + N \times 98.84\% \times a_b && \text{annual costs from false alerts, plus} \nonumber \nonumber\\ & \phantom{+ y[ } + N \times 1.16\% \times a_m] && \text{annual costs from attacks (end quantity).} \nonumber \nonumber \end{align}\)

Importantly, N linearly scales all costs except the initial and ongoing costs (see in Figure 5 (left)), so unless the initial and ongoing costs for a tool are exorbitant, inaccuracies in this file number estimate will not affect any comparison of the tools. For the latter two, we assumed detection was “immediate” using a detection time of \(t = 1\mathrm{E}{-10}\) s.

Fig. 5.

View Figure

5.2 Single-tool Cost Model Results

For comparison in the subsequent results, we include three simulated detector devices: “Never Alert,” which simulates no detector; “Always Alert,” which alerts on every file; and a “Perfect Detector,” which correctly alerts on only malware.

Table 7 displays the average benign-/malware costs and the estimated cost of using each tool alongside the simulated baseline under parameters as shown in Table 6. First and foremost, the simulations provide validation of the model, as all tools perform much better than the always alert simulation (accruing enormous false positive cost but minimal true positive cost) and much worse than the perfect detector device. Because malware is relatively rare (ratio of 1.16%), the cost of never alerting, though still more than all actual detectors, is closer than the other two simulated baselines. The tool rankings show that the host signature tool is superior, because it has no FPs and competitive time to detect. As we only consider the host tools for realistic stand-alone defenses, we note that even with identical initial and ongoing costs, the host signature-based tool is superior. Interestingly, the network static detector places second, because it is by far the fastest detector and has high recall, both outweighing its higher FP rate (high Ave $Benign). The network dynamic tool is hindered by its slow detection time (Figure 3) but benefits from accuracy.

Table 7.

	Benign	Malware	1st Year
Host Sig.	$0.00	$1,435.47	$2,098,712
Host ML	$2.07	$1,440.54	$2,393,151
Network Dyn.	$0.50	$1,500.41	$2,283,562
Network Static	$4.49	$1,089.10	$2,176,719
Never Alert	$0.00	$2,000.00	$2,942,280
Always Alert	$70.05	$350.05	$9,204,530
Perfect Detector	$0.00	$350.05	$549,852

• Cost increases linearly in per-file costs. Mean benignware cost is a multiple of the false alert rate, while respectively mean malware cost is monotonically related to the recall and time to detect via attack cost function.

View Table

Table 7. Cost Model Results (Mean Cost per Benign-/Malware ( \(a_b\) , \(a_m\) ) and 1st Year Cost) with Three Simulated Baselines Using Parameters as in Table 6

• Cost increases linearly in per-file costs. Mean benignware cost is a multiple of the false alert rate, while respectively mean malware cost is monotonically related to the recall and time to detect via attack cost function.

5.2.1 Insights from Varying Cost Model Parameters.

The skeptic might object to cost–benefit analyses, citing that, especially for cybersecurity applications, such methods suffer from required inputs that are fundamentally difficult to ascertain [8, 20], and, indeed, our model is no different. To account for these unknowns, this section investigates parameter sensitivity.

Costs for all tools scale linearly in N, the number of files, so this will not change rankings. Cost increases linearly with mean $benignware with weight \(N\times (1-r)\) and mean $malware with weight r, where r is the malware ratio. Ave$benign is a multiple of the false alert rate, while Ave$malware is monotonically related to the tool’s recall and time to detect via attack cost. Thus, increasing/decreasing r decreases/increases impact of the false alert rate but increases/decreases the impact of recall and time to detect. Precision directly affects results, but recall’s effect depends indirectly on M and \(\alpha\). To investigate, we present costs while varying M and \(\alpha\) in Figure 5. The main takeaway is that the host signature tool is the best standalone choice based on these results and cost model. It offers the lowest cost for nearly all values of M and \(\alpha\), and otherwise (Figure 5) is second to network tools.

5.2.2 Insights from Increasing Cost of “Hard” Files—Zero-days, Polyglots, and APT-style Files.

Another addition to the original cost model that is applicable to this study is increasing the cost of the “hard” malware: zero-days, polyglots, and APT style files, representing the assumption that such files, if undetected, will result in more damage than \(n-\)day files, on average. To exhibit results under these conditions, we compute \(a_{mh}\) the average hard file cost using the detection times on only the zero-days, polyglots, and APT-style functions and with augmented maximum attack cost \(M_{hard}\) (still using the same attack cost equation from Figure 4 caption with identical \(\alpha\)). Similarly, we compute \(a_{me}\), the average “easy” file cost identically on the remaining (\(n-\)day) malware detection results and using a maximum attack cost of \(M = \$2K\). Finally, we replace the final line of Equation (1) (namely, \(N \times 1.16\% \times a_m\)) with a convex sum of the average cost from hard malware files and from easy malware files (namely \(N\times 1.16\% \times p \times a_{mh} + N\times 1.16\% \times (1- p)\times a_{mh}\)) using a parameter p, the percentage of malware assumed to be hard. Results are depicted in Figure 6. The main takeaway is that if one expects (1) more than a couple percentages of malicious files to be a zero-day, polyglot, or APT-style file and (2) the cost from an undetected hard file to be much greater than otherwise, ML-based detectors provide substantial savings.

Fig. 6.

View Figure

6 CONCLUSIONS, LIMITATIONS, NEXT STEPS, AND TAKEAWAYS

This study describes experiments with four prominent malware detection technologies aimed at helping an organization assess (Q1) the ML generalization hypothesis and (Q2) the added value of network-level malware detectors. Our results provide empirical quantification on the efficacy of four market-leading malware detectors. The three ML tools demonstrated a minimal 2\(\times\) increase in detection coverage for publicly available executables and a 10\(\times\) increase in malware detection coverage for zero-day executables. Less intuitive perhaps was how well the host signature–based tool performed: exhibiting the best (perfect) precision; demonstrating comparable recall (\(\hbox{$\scriptstyle \mathtt {\sim }$}\)35%) to the host ML–host tool; and, according to the cost model, performing the best overall for small percentages of hard (zero-day, polyglot, or APT-style) files and when maximum possible damage cost incurred by an undetected hard file was not much more than an n-day malware. Notably, if one expects (1) more than a couple percentages of malicious files to be zero-day, polyglot, or APT-style files, and (2) the cost from an undetected hard file to be much greater than otherwise, then ML-based detectors provide substantial savings; this provides a strong argument for pairing detectors. Combining the signature-based tool with an ML-based tool enables malware detection ranges above 90% for in-the-wild executables and above 60% for zero-day malware. As demonstrated by increased recall and the new configuration of the cost model, substantial value is added by combining either of the two network-level tools to either host-level tool. Unsurprisingly, dynamic analysis incurs the cost of latency for increased recall, and both network-level detectors vary in their capabilities across protocols. Our results indicate that low false positive rates (precision \(\gt\)98%) should be expected, with fast detection time for static detectors, but that this comes at the cost of recall. Of the four detectors in this study, none surpassed 55% recall, and recall falls dramatically with JAR and polyglot file types.

Testing only four representative tools is a clear limitation; in particular, many COT tools offer (and some require) cloud connectivity to enhance detection at the expense of privacy, but no such tool was represented. Our test environment did not allow connection to the Internet, which might have stymied malware actions and therefore affected detection results. Similarly, network-level signature-based tools are not fully represented. Impact to host resources was not taken into account in this study (CPU, memory, and disk I/O, in particular). More sophisticated (and presumably more accurate) models of attack cost could be integrated in future work (e.g., Reference [23]). Finally, accurate cost estimates from the cost model will require per-SOC inputs. Based on our parameter sensitivity analysis, the rankings and takeaways of the cost model results for these tools should generalize (and our treatment provides a blueprint for future use). Nevertheless, we argue that trends found among these four market leaders are sufficient to pose hypotheses, raise awareness of gaps, and sharpen next-step research.

We itemize priorities for next-step research from this work: Our polyglot detection results suggest that detectors are failing at preclassifying a file into possibly many file types for which it can execute. Research is needed to accurately and quickly determine all of the file types of a given sample and integrate this capability into COTS detectors to enhance detection. Further research is needed to address the polyglot detection issue and performance with throughput and computational expense in mind. More accurate detection models for all file types are needed. In particular, future malware detection research should focus on increasing the detection capabilities for non-PE and non-MS Office file types. This might entail enhancing the quantity of malware and benignware files of many types that are currently less prevalent (e.g., through an effort to build and make public a more diverse malware data set). For the two network-level detectors, recall varied widely when identical files were delivered on different protocols. We suggest a study to identify whether the file carving is the limiting factor and to find methods for accurately extracting a copy of the file from the packet stream to the detector. Future work to tune the cost model to a specific SOC is both reasonable and likely very valuable if successful, especially if it produces a procedure and code-base for refitting to any SOC. Malware detection studies in which computational resources of the hosts are monitored and incorporated are needed, in particular for a higher-fidelity cost model. As evidenced by our results, many of the takeaways depend on difficult assumptions; for example, the percentage of files that are malware (e.g., Li et al. [24]) or the percentage of different file types or categories in the wild. Empirical studies to provide data on cyber-related artifacts provides fodder for useful studies.

Finally, some takeaways for SOCs. Great gains in detection rate (recall) with few false positives (high precision) are permitted by all pairs of tools in our study, but little is gained from using more than two of these tools. When choosing a pair of tools, strive for tools that complement each other in terms of increasing the coverage (detection rate) of all file types, rather than duplicating results—this might require some preliminary testing with different file types. A feasibility study for any desired pairing of host tools is needed before deployment to examine how the combination will affect the host’s resources. Our results (Table 8) show that substantial savings are yielded by adding a network-level detector in all four combinations of network-to-host detectors tested. The accuracy gains permitted by the dynamic detector in our study outweighed latency costs. For any network detector, testing bandwidth and file rates with the specific software and hardware is needed. Note that network-level detectors require unencrypted traffic, so we are presupposing that an SSL/TSL “break and inspect” technology is in place.