Black Box Analysis of Android Malware Detectors

Black Box Analysis of Android Malware Detectors by Guruswamy Nellaivadivelu Code obfuscation can make it challenging to detect malware in Android devices. Malware writers obfuscate the code of their programs by employing various techniques that attempt to hide the true purpose of the program. Malware detectors can use a number of features to classify a program as a malware. If the malware detector uses a feature that is obfuscated, then the malware detector will likely fail to classify the malware as malicious software. In this research, we obfuscate selected features of known malware and determine whether the malware can still be detected by a given detector. Using this approach, we show that we can effectively perform black box analysis of various malware detectors.


Introduction
The volume of Android malware is increasing exponentially. Indeed, in the second quarter of 2016, 3.5 million samples of Android malware were detected [1]. This rapid increase in Android malware has placed the focus on Android security and made it imperative to develop more efficient defensive tools for combating such malware.
One of the challenges faced in this area is the use of code obfuscation techniques.
Code obfuscation is a method of altering code to hide its actual purpose, without significantly altering its performance. There are many ways of obfuscating source code in an Android environment. Several software applications that are available off the shelf can be used to achieve different levels of code obfuscation [2]. In order to address the problem of strengthening malware detector's strength, there are two fundamental questions that need to be addressed, as highlighted by Christodorescu et al. [3]. The first question is to gauge the resilience of a malware detector against code obfuscation. This will also help us in understanding the strength of the malware detetctor in detecting variants of known malware families. The other question is the possibility of identifying the detection algorithm used by a malware detector.
By studying the behavior of malware detectors and how they respond to different obfuscation techniques, a malware writer can uncover ways to beat the antivirus program. Ultimately, we want to gauge how well a malware detector will perform against obfuscated code.
Code obfuscation is the process by which source code is manipulated to hide its true intentions. Code obfuscation is increasingly becoming a common tool to avoid detection by traditional malware detectors. There are many different types of code obfuscation. The most basic type of code obfuscation involves the encryption of all the strings that are used in the code. This overrides the detection mechanism of most of the traditional malware detectors. Some advanced malware detectors account for this encryption and are able to identify malware files. There are a host of other obfuscation techniques that can be employed by malware writers. Some of these include the obfuscation of function calls, permission hiding, and insertion of dead code.
The challenges associated with code obfuscation primarily deal with the problem of maintaining the core functionality of the code, while making it difficult for malware detectors to detect their true purpose. This challenge becomes easier for malware writers when dealing with Android malware. The reason for this is associated with the permission levels of applications running on Android platform. Unlike antivirus programs that run on computers, the Android system provides the same set of permission levels to the anti-virus application and the application that is being scanned. This is a major limitation for malware detector writers. With the advent of sophisticated encryption techniques, it has become very difficult to different between benign and malicious applications of obfuscation techniques. The primary objective of this project is to make malware detectors more responsive to the code obfuscation techniques employed by malware writers. By doing so, we can attempt to identify the malware features that are used by a malware detector in its classification algorithm.
We can also try and modify an existing malware detector to see if we can overcome the limitations. In order to achieve this, we propose a theoretical approach. In this approach, we attempt to isolate the features that contribute to malware detection.
Once, we have this information, we can attempt to modify an existing malware detector to overcome these limitations. The malware detector should employ ''de-obfuscation'' techniques before analyzing any malware. An intelligent malware detector should be able to sense the type of encryption or obfuscation technique being employed and use the corresponding ''de-obfuscator'' to nullify the effects of the obfuscator.The first step in this implementation will be the identification of the factors in a malware that are taken into consideration by a malware detector. To achieve this, we will begin by encrypting various parameters of a malware and running it through a malware detector [4]. By following this approach, we can identify the exact scenario when a malware is no longer classified as a malware by our malware detector. Once we identify the features that are required by a malware detector, we will use this information to make the malware detectors process the obfuscated part of the code as well. This will make our malware detector more robust and improve their performance.
In Chapter 2, we look at the previous work that is done with regards to malware detection in Android. We explore the various detection mechanisms and approaches that has been discussed so far. After looking at the background work, we delve into code obfuscation in Chapter 3 and understand the basic terminologies associated with code obfuscation. We also look at the impact of obfuscation in general, and then more specifically, their impact on malware detectors. After understanding the basics of code obfuscation and malware detectors, we move on to the current threats and defenses in the Android operating system in Chapter 4. In Chapter 4, we glance at the growing dominance of the Android operating system in the mobile phone space and the importance of this particular operating system in our lives. The motivation behind selecting the Android OS for this project is understandable from Chapter 4.
The obfuscators to be used in this project, and their functionalities are explained in Chapter 5. Chapter 6 clearly lists the software requirements for this project and also talks about the necessary technologies for setting up the experiment. The results of the experiment are summarized in Chapter 7. The factors that contribute to the conclusion being drawn from this experiment are detailed in Chapter 6. We finally consolidate the results and discuss the future course of the project work in Chapter 8.
In this chapter, we present the results of a literature survey that was performed to identify the current state of obfuscation mechanisms and their impact to the field of code obfuscation. We find that code obfuscation has been an area of interest in the field of cryptography and traditionally, obfuscation techniques have been used to achieve reverse engineering protection. On the other hand, a lot of malware have obfuscated code to avoid detection by anti-virus programs.

Code Obfuscation and Malware Detectors
The efficiency of malware detectors against code obfuscation has been a point of discussion amongst malware researchers for a very long time. A lot of research has been done on the robustness of malware detectors against high levels of obfuscation. The issue of malware detector's strengths against obfuscated malware had been discussed as early as 1996, as can be seen in the quote by S. Gordon and R. Ford [4]: ''The evaluation of anti-virus software is not adequately covered by any existing criteria based on formal methods. The process, therefore, has been carried out by various personnel using a variety of tools and methods.''

Program Obfuscation
There has been a lot of theoretical research on the different aspects of obfuscation and on ways to improve it. Most of this research has been successful in arriving at a conclusion on the efficiency of the cryptographic problems of encryption, authentication and protocol [5]. But the problem of program obfuscation has remained an area within cryptography in which theoretical research has been inadequate. In their seminal paper on program obfuscation, Barak et al. [5] propose to represent program obfuscation as below: An obfuscator is said to be an efficient compiler if it takes as input a program and produces a program ( ) and satisfies the following two conditions: 1. Functionality: ( ) computes the same function as 2. 'Virtual Black Box'property: Anything that can be efficiently computed from ( ) can also be computed by .
The paper by Christodorescu et al. [3] lists various ways to test and achieve program obfuscation in general. A detailed analysis of the various obfuscation methods is also discussed in the paper. One interesting angle explored by the paper deals with assigning mathematical equations to measure the effectiveness of the individual obfuscators. This lets us quantify the different obfuscators and rank them against each other. One of the evasion methods employed in malware obfuscation is polymorphism.
It is a method by which a program evades various detection tools by mutating into different forms. In the paper by Rastogi et al. [6], the authors develop and propose a framework called 'DroidChameleon' that provides a way to transform Android applications into different forms with minimal user involvement. As shown in Figure 1, the authors apply various transformations on a malware sample dataset. The output of all these transformations are processed by a malware detector (referred here as Anti-malware). The input to the anti-malware is processed sequentially. After each Figure 1: Evaluating anti-malware transformation, the anti-malware's output is evaluated and if the malware detection fails, the next level of transformation is applied. This helps rank the various malware detectors against each other for accurate analysis.

Obfuscation in Android Malware
A report by Google stated that a majority of malware detectors work as a binary classifier [7]. They classify an application as a malware or a benign file. In order to effectively eliminate malicious applications, it is important that malware detectors do more than just identify malware. They should be able to isolate the core parts of the application that perform the malicious acts and work at fixing the loopholes that let the program act in a malicious way. More recent malware applications employ a variety of tricks, in addition to traditional code obfuscation mechanisms. For instance, a variant of Android malware, known as Android/BadAccent, is a known banking Trojan, that steals credentials used in banking applications [8]. A variant of this malware used a mechanism known as 'Tapjacking' to extract the credentials from the users. In this form of attack, a screen is displayed to the user, while a second screen is hidden behind the actual visible display [9]. When a user clicks a button on the screen, assuming it to be the one that is displayed, the underlying screen gathers the input and processes the command. This is a common method of gathering details from unsuspecting users.

Statistical Anaylsis Techniques and Android Malware
One widely used approach for analyzing malware samples is the usage of statistical methods. In such methods, the Android executable file (with the extension apk), is decompiled to get the original source code. Due to the Android operating system being written in Java, it is easy to reverse engineer an apk file to retrieve the source code. This opens up many opportunities for performing statistical analysis on the obtained raw data. This also lets a researcher perform various operations on the source code, and then repackage it back into an apk. In the approach known as AndroSimilar, Faruki et al. [10] propose a new algorithm known as AndroSimilar, that takes into consideration various features that are known to be present in malware alone. The AndroSimiar approach [10], as shown in Figure 2, decompiles an apk file and repackages it after feature extraction. To extract the features, the algorithm incorporates apps from the Google Playstore and other third party applications. These features are normalized and fed into a signature generation engine, that provides a unique signature for each malware. This is used as reference for detecting future malware applications.

Conclusion
Malware in mobile devices is no longer a problem confined to labs and research areas. The rapid increase in access to computers has helped malware writers create specific, targeted programs that perform with high efficiency and exploit vulnerabilities in different operating systems. The amount of research being done in malware analysis and, more specifically, in Android malware, is in the right direction. In the fight against sophisticated metamorphic malware, it is imperative that the malware detector is better than the malware creator. In this paper, we have explored various work, that dealt with the different aspects of malware obfuscation and ways to overcome the shortcomings in today's version of malware detectors. The future of malware looks very bright and it is hoped that the malware detectors of the future will be up to the task at hand.

CHAPTER 3 Code Obfuscation
Code Obfuscation is a technique by which programmers have deliberately sought to make the functionality of their code less obvious. This technique has been used by programmers to achieve various additional objectives. Code obfuscation can be used to achieve a myraid of objectives. These include prevention of reverse engineering, protection of intellectual property, and reducing the size of an executable.
In some benign scenarios, an executable is obfuscated to protect the various licensing mechanisms used in them. Obfuscators are also a good way to restrict unauthorized access to files by people who might try to use dubious tools to incorporate malicious code into files.
We will look at the history of code obfuscation to appreciate the relevance of code obfuscation in today's software development perspective. With growing interest in various obfuscation techniques, and the ease of availability of obfuscators dedicated to different operating systems, this would help us in understanding the rapid growth in this area and appreciate the urgent need for various countermeasures against this approach.

Growth of Obfuscation in Software Development
Code obfuscation has been historically associated with malware development, than with benign software development. Some of the earliest examples of attempts at obfuscation in malware can be found in the ''Brain Virus'' . In this variant of the malware, the malicious program would display unaffected disk partitions to users attempting to access partitions that the virus had corrupted. Although the code in itself was not encrypted, the behavior of the virus shows attempts at hiding its true usage.
In the same year, the Cascade virus was released to the world. This was an early variant of malware to use encryption to hide its true purpose. The earliest strains of obfuscated malware used a simple encryption-decryption routine to perform the decryption tasks. As the malware detectors of the time were not sophisticated enough to detect the encrypted part of the code, this simple obfuscation technique enabled a lot of malware programs to slip away undetected. This is a serious disadvantage in the design and implementation of malware detectors. We would be exploring more such flaws with the implementation of malware detectors in this project.
With the advent of advanced malware detectors and improvement in statistical analysis techniques, the level of obfuscation in malware increased. Polymorphic malware uses a very high level of encryption technique to obfuscate its contents. A polymorphic malware changes the encryption in itself and provides very few traces of a signature. If a malware is truly polymorphic, then there will be no consistency between any two iterations of the same program and it would be virtually impossible to detect them using traditional signature matching techniques.

Malware Detectors
Malware detectors came into existence with the advent of different malicious programs. Before the rapid growth of the internet, malware detectors were only capable of performing scans based on signatures of known virus programs. This static analysis technique meant that new virus would be out in the wild for some time before the malware definitions of the individual anti virus programs could be updated. With the introduction of the world wide web, the antivirus industry expanded into dynamic analysis and cloud based malware detectors. Firewalls, online scanning, and virtual machines started being increasingly used to identify malware. One major shortfall of anti virus programs is their inability to detect polymorphic virus. In general, many antivirus programs employ signature detection for identifying malware. In addition to this most common approach, heuristics based detection and rootkit based detection are also employed to detect virus programs. Along with these approaches, active scanning approaches like on-access scanning is also used to detect programs that might attempt unauthorized operations. We discuss these methods and detection mechanisms in detail in this chapter.

Signature Based Detection
This is one of the most basic methods of malware detection that is still in use today. When a new strain of malware is detected in the "wild", antivirus firms analyze it and extract a "signature" from it. This signature extraction can either be done manually or by using automated signature detection techniques [11]. Once a signature is detected, it is updated into various malware definitions of antivirus software. Although this method is effective against generic malware, it is highly ineffective against oligomorphic, polymorphic and metamorphic malware. These are variants of malware that encrypt itself with each iteration. In this project, we attempt to identify the various factors that contribute to malware detection and their importance in overcoming the signature detection method.

Heuristics Based Detection
In Heuristics based detection techniques, a single signature or pattern is used to detect multiple malware belonging to the same family. Such techniques rely on the fact that multiple malware are created from a single malware. Thus, successfully creating a signature for a base family will result in the detection of all malware related to that particular family.

Rootkit Detection
A rootkit is a type of software that attempts to gain administrator privileges in a system without the knowledge of the user running it. In many cases, the rootkits contain software within them that becomes undetectable to antivirus programs.
Rootkits usually have full administrative access and also have the ability to hide themselves from the list of running processes. Modern antivirus software scans for rootkits in specific, to detect them. It is very difficult to remove a rootkit when compared to other generic malware programs.

On-Access Scanning
In this method, the antivirus program looks out for any threats that might happen on a real-time basis. The antivirus monitors the system in which it is installed and looks for suspicious activity whenever the computer's memory is loaded with fresh data from the storage disks. This might happen when a USB drive is inserted, an email attachment is opened or a even when an already existing file is opened by a user or a program. This type of scanning is more effective as it does not rely solely on malware definitions to detect viruses.

CHAPTER 4 Threats and Defenses in the Android Operating System
Before we dwell deep into code obfuscation in Android, we look at the various malware detectors for the Android operating system. We also look at the rapid proliferation of the Android OS and the reason for selecting Android as the focus of study in this experiment.

Android Malware Detectors
With the rise of the Android Operating systems, the amount of malware associated with it has also risen significantly. From a market share of 2.8 % in 2009 [12] , Android captured about 75% of the market in 2012 [12]. As shown in figure 3 , we can see that the growth and adoption of Android has been very steep. This rapid proliferation of Android resulted in an equally rapid rise of Android malware.

Privilege Escalation
In this type of attack, the malicious app that is installed on a device, attempts to grant itself additional privileges than the one it requires. This is achieved by using known exploits in the Android operating system.

Remote Control
A very high percentage of malware attempts to use the compromised device as a remote bot. In some malware families, the remote URL that is being used to control the device is encrypted. Such encryption makes it very difficult to detect these types of malware and this will be a primary area of focus in this thesis.

Monetary Loss
A very direct way of monetizing malware is to make unsuspecting users subscribe to services that cost a lot of money. Such services are run by the malware perpetrators and will enable them to charge the infected devices' owners money for services that they are not aware of. To achieve this, some malware use the remote control to push down numbers of services to the devices and then enroll them.

Information Collection
Many malware programs attempt to collect the personal information of users.
Such personally identifiable information makes it easy for scamsters to dupe people using various other schemes. Malware belonging to this family tries to steal personal information of the compromised device's owner, as well as the details of people in their contact lists. This information is then sold through different means to interested parties.

Android Malware Detection Limitations
One of the major limitation of malware detection in Android is the limited processing power of the devices running Android. Due to processing and memory constraints, generic malware detection has to be restricted to static analysis techniques.
In general, all the existing Android security solutions can be classified into Static Analysis and Dynamic Analysis [14].

Static Analysis
Static Analysis is a technique in which the an application is evaluated for its trustworthiness by disassembling and checking its source code. The application is not executed for this analysis. Once an application is marked for scanning, various statistical analyzing approaches are used to classify the file. Some of the most commonly used static detection methods are discussed in the next few subsections.

Signature Based Detection
Signature based detection is a type of static analysis technique. In this method,

Permission Based Detection
This is a straightforward approach to detecting malware in Android systems. In this method, the number of permissions an application requires is used to determine its classification as a malicious or a benign file. Some research has been done in this area wherein the Android Manifest file is analyzed for extracting information [15] about the permissions requested by the application. This information is used to assign a score of relevancy to the permissions requested. This score is then compared against a threshold for determining the malicious intent of an app. There are variations to this technique and some methods yield better results than the others. This method is a very quick way of determining the malicious nature of applications. But a serious limitation of this method is that it does not analyze the source code or the working of the app. Only the Manifest file is analyzed. A lot of malware apps use permissions similar to the benign apps. Hence, permissions based detection should be used in conjuction with a second confirmation method to validate an app.

Dynamic Analysis
In this method, the application is executed and it is analyzed during the runtime.
It becomes very easy to identify sections of code or execution blocks that were missed during the static analysis of an application. Dynamic analysis methods are also effective against obfuscation and encryption techniques.

Anomaly Based Detection
An application is executed and the system calls generated by it are recorded in a log. This log is then sent for analysis to a remote server, where the various behavior of malware are recorded. Using that as a basis, the log files are analyzed, and the results are aggregated. This result, in collaboration with other techniques are used to classify the file as malicious or not.

Emulation Technique
Yan et al. [16] propose a technique in which a virtual machine is used to analyze an application. In common virtual machine based detection techniques, the antimalware program and the malware execute in the same environment. This makes them detectable to each other. In the platform presented by Yan et al. [16], the antimalware, DroidScope, stays out of the execution environment and monitors the execution as a whole. This enables it to detect the malware without being detected by the malware.

CHAPTER 5 Android Obfuscators
In this chapter, we use different obfuscators to modify parts of an android malware.
By systematically obfuscating different parts of the code, we can gain insight into the parts which contribute most to the detection of malware. Once we have this information, we can then determine efficient ways to make the malware detectors more robust and be less resilient to code obfuscators.

Experiment
For this project, we use a tool called AAMO (Another Android Malware Obfuscator) [17]. This tool gives us various obfuscators for use with our experimentations.
The obfuscators can be used independently or in combination with other obfuscators to increase their effectiveness. Using this tool, we decompile a android file, perform obfuscation operations on them, and recompile the file again. In this experiment, we use the source code provided by the developers of AAMO [17] and available at [18].
This tool forms the basis of the work presented in this thesis. The steps involved in this are detailed below: 1. Obtain an APK file.
3. Get the list of obfuscators passed into the program. 4. Apply the obfuscators one after the other on the decompiled apk file.
5. Repackage the decompiled file into an APK.
6. Sign the APK file to maintain its integrity. Performing the above steps ensures that the apk file is not corrupted and its usage is not affected. We perform this to make it difficult for a malware detector to detect the apk file as a malicious one. The entire flow of the experiment is depicted in Figure 5.
As shown in Figure 5, the final encryption would let the malicious file be signed with a valid signature and thus eliminating any traces of the apk file having been compromised.

Uses of the obfuscator
Using the obfuscator in this step has various advantages for our experiment. One of the primary uses is to make the job of the malware detector more difficult. Since most of the malware detectors do not take into account polymorphic and oligomorphic malware, using obfuscators will let us know which parts of a malware factor into the detection score computed by individual detectors. In this experiment, we use 14 obfuscators to test out the resilience of the malware detectors as listed in Table 1. These obfuscators enable us to test the various aspects of a apk file and help us determine the ones that are really useful to a malware detector. When a particular obfuscator is run, it runs a function that is specific to that particular obfuscator and applies that function to all the parameters that match the criteria for that specific obfuscator. Each of the obfuscator is discussed here in detail.

Resigned
This obfuscator decompiles an apk and just resigns the apk file after compilation.
Not much change is done to the application file in itself. The purpose of this obfuscator is to attempt defeating malware detectors that try to use signatures of certain known malware sources to classify a malicious file.

Alignment
This obfuscator makes use of the zipalign utility of android. Zipalign is a tool that is used to provide optimization techniques to APK files. The tool causes all uncompressed data within the APK to start with a particular alignment relative to the file's beginning. The Alignment obfuscator changes this alignment before recompiling the apk file.

Rebuild
This obfuscator rebuilds the application file without performing any changes. The unpacking and repackaging of the apk file affects the timestamp, signature of the apk and other factors that help in identifying the origin of the file. Some smart malware detectors are able to detect these changes and do not let the file pass through it.

Fields
This is a relatively simple obfuscator that just renames the fields that are used in the application. This is done after the decompilation of the apk file. The smali is analyzed for locating the fields that are used in the source code and these are renamed.

Debug
The debug obfuscator removes all information related to debug from the files.
This is performed not only on the smali file, but throughout the source code as well.
Without the debug information, the APK file becomes slightly different from the original file. Removal of the debug information also alters the size of the file and makes it different.

Indirections
Call indirections is an advanced obfuscation method in which various function calls are directed through different values. The obfuscator performs operations such as changing the register count, changing a method call and also redirecting all calls to the methods. This obfuscation completely changes the control flow of an application and makes it difficult to detect using a comparison model in dynamic analysis as well.

Renaming
All the variables in the sourcecode are renamed to different values. This is exactly like using substitutions to hide the original values. Renaming is also advantageous when certain signature and pattern matches are based on the names of the variables and functions.

Reordering
Using reordering will let us change the order of the code in the application. The obfuscator changes the location of certain parts of the code and adjusts the calls to it accordingly. This makes it possible to evade signature based detection methods if the signature is based on the order of instructions or if it is based on the DEX opcodes.

Goto
In order to modify the control-flow structure of the application, forward and backward jumps are inserted into the code. These unconditional jump statements will be executed irrespective of how the program is run. This widely alters the flow and will make it very difficult to detect using conventional methods.

Arithmetic Branch
A constant value, known to the obfuscator, is used to achieve this obfuscation.
This constant value is not known to the compiler. Using this constant value, the obfuscator is able to control the flow of execution of the program. The compiler assumes that either of the branches could be possible as the value for deciding the flow of control is not known. This is applied to methods with more than 2 parameters.

Nop
This is a classical and an easy way to obfuscate a program. In this, a "no-operation instruction" (known as a "NOP") is inserted into the source code. The number of such instructions inserted is randomized. These are inserted into methods to make them bloated and delay the execution time.

Lib
MD5 hashing is used to rename the file and path names. A proxy method is created and used to handle the decryption of the values, when it is required by the system.

Manifest
The AndroidManifest.xml file is modified by this obfuscator. The manifest file contains important information related to the application's usage and permissions.
This obfuscator opens up the file and encrypts the values for the resources and also replaces the characters in user defined identifiers.

Reflection
This obfuscator acts similar to the code reodering obfuscator. The reflection obfuscator takes advantage of the Android dynamic code loading API. All the static method calls are converted into reflection calls and the the reflect method is invoked on a string that contains the target method's name. Due to the various different types of software used in the experiment, it is important to have the correct version of each software installed. As shown in Figure 5, each APK will have to be decompiled into its source code, before going through the obfuscation process. To achieve this, we use a program called apktool [19]. A list of various software and their versions are listed in Table 2. The decompilation of the APK files is achieved using the Apktool. Apktool requires a java virtual machine to execute. It is imperative that this version of Apktool be maintained for repeating the experiments presented in this work as the source code of AAMO has been modified to fit this version of the tool.

Dataset and Malware Detectors
In order to successfully evaluate and analyze the malware detectors various experiments were performed using known Android malware. Once the android malware were finalized, the obfuscators were chosen to increase the difficulty of malware detection. A sampling was performed with a handful of malware. Using this sampling, the obfuscators to be applied were selected and then applied to a wider dataset. The  Figure 6. In Figure 6, a sample file, ''Angry Birds'', was used to test the effect of the various obfuscators. This file, a malicious version of the popular game, is a Trojan variant that steals the contact information, and has the ability to send text messages without the user's permission. Before applying any obfuscator, the file had a detection ratio of 0.819. When we apply the obfuscators, the detection ratio drops steadily. In Figure 6, the x-axis represents the different obfuscators that were used. We can see that the detection ratio almost remains constant for all the obfuscators, except for the manifest and reorder obfuscators. The functioning of these obfuscators are defined in chapter 5. This hints at the fact that many malware detectors just perform an analysis on the AndroidManifest.xml file to classify the file as a malware. Due to this, when the manifest obfuscator is applied, the detection ratio drops. When we apply all the obfuscators on the file, the detection ratio drops significantly. This is shown in the Figure 6's 'allObfuscation' bar.
It can be seen from the results that only some obfuscators contribute effectively to hindering the detection ratio of malware obfuscators.

Dataset
The Contagio dataset was used to perform the various experiments in this project [20]. All the samples used for experimentation are malicious files. The files were classified as malicious by various means and the contagio data dump also certifies the files as being malware.

Malware Files Selection
Known malicious files were used for performing the experiments in this project.
The reason for using malware for the experiments was to understand how each obfuscator would help the malware in evading detection by malware detector. All the test samples were caught by at least one of the malware detectors and many of the samples were incorrectly classified as benign files, once the obfuscation was complete.

Other Datasets
Previously, experiments have been performed on malicious files belonging to other datasets. Before we delve into the results of the experiment performed in this work, we look at how obfuscators affect the detection ratio of various malware detectors.

Malware Detectors against Code Obfuscation
A single malware detector is unlikely to give us a substantial result. This is because various malware detectors use different techniques for analyzing malware. If a single malware detector were to be used as a benchmark, then we would either get excellent detection scores or the malware detector would fail in a very poor way. To overcome this shortcoming, a single obfuscated file is scanned by several malware detectors simultaneously. Instead of manually uploading the files to different malware detectors, we make use of VirusTotal [21] and other similar virus scanning providers.

Results of Experiments
The various obfuscators defined earlier were iteratively applied to malware samples from the Contagio dataset. Based on the results obtained from VirusTotal, the obfuscators were selected for further application.

Observations
The VirusTotal website uploads a malware file to its database and then performs a scan using the various malware detectors associated with the website. Each uploaded file is hashed and stored in the database to reduce duplicate efforts and minimize scan times. Due to this behavior, each time a file is loaded into the website to be scanned, the website will prompt if a similar file was scanned earlier. It was observed that as the number of obfuscators employed increased, the similarity between the obfuscated and un-obfuscated applications decreased. If more than 2 certain obfuscators were applied, the VirusTotal website would not recognize the file as a previously recognized file. This observation was consistent throughout the different experiments conducted.

Steps for Analyzing Malware Detectors
The experiment was performed with certain operations being repeated in an iterative manner. The obfuscated malware files were prepared in advance. The steps are as follows: • Scan a malicious file using VirusTotal.
• Record the detection ratio.
• Apply obfuscator(s) on the selected malware file.
• Scan the obfuscated file using VirusTotal again.
• Record the new detection ratio.
Repeating the above steps helped us detect how robust and efficient malware detectors are. Ideally, the malware detector should not be affected by the obfuscators.
The detection ratio should not be very different irrespective of whether the malware was obfsucated or not.
But the results indicated that almost all the malware files had a very high probability of being classified as a benign file, if they had sufficient obfuscation techniques applied to them.

Metrics used
We use the detection ratio provided by VirusTotal to determine the effectiveness of the Malware Obfuscators. As expected, the malware detectors are not resilient enough to detect variants of malware that have been slightly obfuscated.

Obfuscation of Malware Samples
The results for applying each obfuscator were collected and only the significant results are shown here. In addition to gathering the results for an individual obfuscator, we also get the results for the individual malware detectors. A comparison of their behavior is also presented here.

Individual Obfuscators
Application of individual obfuscators did not alter the detection ratio by a huge margin. A sample detection ratio for applying the "Renaming" obfuscator is shown in Figure 7. As part of this experiment, 289 files were obfuscated and run through the malware detectors. In Figure 7, the malware samples are represented on the x-axis as alphabetic symbols. The mapping for the malware file to symbols is shown in Appendix B. The graph shows that the average detection ratio for the renaming obfuscator is 0.46. The average detection ratio is represented by a red line in the graph. While there are some occasional spikes in the detection rate, that seems to be the exception with a very few files being consistently classified as a malware.
The results for the obfuscator "Manifest" are shown in Figure 8.The detection ratio for this obfuscator is also lesser than for a normal malware. But with applying this obfuscator, the detection ratio is further reduced. The average detection ratio in this case is 0.3867. This shows that the manifest obfuscator contributes more to the detection rate. We repeat this experiment for different obfuscators to get record their detection scores. These are included in appendix C.

Multiple Obfuscators
While individual obfuscators didn't provide much insight into the malware detection scores, it was observed that combining multiple obfsucators quickly decreased the detection ratio.
In Figure 9, the obfuscators Renaming, Reordering, Goto, and Arithmetic Branching were applied to the files.
This certainly increased the obscurity of the malware files. The detection ratio for the obfuscated files in Figure 9 is much lesser than in Figure 7. This could be attributed to the fact that a combination of weak obfuscators is still a strong enough challenge for malware detectors. We also note that, with a average of 0.4034, this is only marginally better than the performance of a single manifest obfuscator as shown in Figure 8. This further reiterates the significance of selecting the right obfuscator rather than a combination of different obfuscators.

All Obfsucators
To make the results of the experiment certain, all the obfuscators in question were applied to a set of files. Keeping up with the consistency observed so far, the detection ratio dropped by a huge margin. This can be observed in Figure 10. The results show a average detection rate of 0.21. This is extremely low when compared to the results obtained before any obfuscation was applied as seen in Figure 6.
This clearly shows that by increasing the number of obfuscators being applied to a malware, we can bring down the detection ratio of that particular file to a very low value.

Average Ratio and Summarization
To conclude the experiments, the average detection ratio was calculated for each obfuscator and the combination of obfuscators. The results of this calculation are shown in Figure 11.
We observe a decline in the detection ratio for the different obfuscators. The average detection rate drops steeply when all obfuscators are combined. This is consistent with the results obtained so far in the experiment. The presence of the manifest obfuscator after the result of the combined four obfuscators in Figure 11 shows the importance of selecting the right obfuscator for defeating the malware detectors.

Individual Malware Detectors
We now analyze behavior of the individual malware detectors and look at their performance against code obfuscation. To achieve this, we perform the same experiments as before and gather the detection statistics for each malware detector. This analysis will let us understand the workings of a particular malware detector and help us identify the best detector for Android. Once we have that information, we will know the best way to defeat obfuscation in malicious programs.
The detection rates for the AVG Antivirus are shown in Figure 12.   The TrendMicro Antivirus performs well only against unobfuscated malware samples, as seen in Figure 14. The performance of this antivirus is consistently poor across all the obfuscators. We can surmise that the algorithm being employed by the TrendMicro antivirus is not very effective against obfuscated Android malware.
It is safe to conclude that the detection algorithms of AVG and BitDefender are much more efficient against obfuscated malware, than the algorithm used by the TrendMicro program.

Conclusion
Due to the limited processing capacity of the mobile devices, it is imperative that stand alone malware detectors are able to sufficiently defend against known threats and variants of known malware that are detectable by signature scanning. In this experiment, we used different obfuscators to test the resilience of malware detectors against obfuscated malware. Unsurprisingly, the malware detectors fared very poorly against such obfuscation techniques.
We also observed that by applying all the obfuscators, it is becomes a trivial task in [17]. The conclusions drawn by them indicating a huge gap in the requirement and the availability of sophisticated anti-virus products is still very much prevalent.

Future Work
Similar to the obfuscators employed in this experiment, it should be possible to create ''de-obfuscators'' for Android files. It would be interesting to see the effect of each de-obfuscator against the corresponding obfuscator that has been used here. If employing such a de-obfuscator helps in thwarting the obfuscation, then it could form the basis for developing more generic de-obfuscation algorithms for incorporation into malware detectors. As was evident from the experiment, the selection of the right obfuscator could greatly influence the detection rate. This proves that the majority of the malware detectors place too much of significance on one aspect of a file, for classifying it. This shortcoming with the malware detectors for Android should be taken care of. With these conclusions, we hope to make future malware detectors more resilient against polymorphic virus with the expectation that the creators of the anti virus software incorporate the necessary changes to their programs.

Appendix 1 A.1 Abbreviations and Terminologies Used
• APK -A file format used for installing applications on the Android operating system.
• AAMO -Short for Another Android Malware Obfuscator -A program for applying obfuscators to android files.
• Obfuscator -A small program that scrambles the source code of a program to make its functions less obvious.
• Apktool -A program that is used to decompile and compile APK files.