An Approach for iOS Applications’ Testing

. The mobile conquers the world. The need in a comprehensive and systemized multi-edge testing approach is rising along with mobile apps becoming even more complex. As a leader in enterprise market, Apple iOS has been chosen as a target mobile platform for the study. The authors have investigated aspects that influence functional testing of iOS apps in particular, and mobile – in general. The study also exposes the security capabilities and risks that often are not mitigated in favor of time to market rush. Investigation and clustering of mobile UI test automation tools is performed. The capabilities and limitations of Apple UIAutomation are discovered. Solutions aimed at overcoming the limitations of out of the box UIAutomation are united in tTap framework developed by the authors. In conclusion an ideal cross-platform mobile UI test automation tool is proposed.


Introduction
In 1975, the first theoretic foundation of testing by Goodenough and Gerhart (1975) was published.A year before that the first publication on software testing was published in Latvia by Barzdin et al. (1974), but an enriched version of it was presented in 1977.Since those times theory and practice of testing have evolved quite significantly through emergence of testing activists (Myers (1979(Myers ( /2004)), Beizer (1990Beizer ( , 1995)), Kaner (1999Kaner ( , 2001)), Bach (Kaner et al., 2001), Pettichord (2007), Kaner et al. (2001), Black (2009), etc.) and under the influence of different software development approaches (waterfall, rapid application development, agile, etc.).Nowadays, testing has become a crucial part of the software development process.
The rise of mobile technology has touched upon the lives of everyone.According to the study by Research and Markets (2014), the mobile cloud market is expected to be worth US $46,90 billion by 2019, while the research by Markets and Markets (2014) shows that heterogeneous mobile processing & computing market will be worth US $61,70 billion by 2020.iOS from Apple is one of the most popular mobile operating systems.According to Citrix (2015), iOS holds 64%, and according to Good Technology (2013), iOS holds even 73% market share of all enterprise mobile devices.According to the same study by Good Technology (2013), iPads hold 91,4% of enterprise tablets.That is why iOS has been chosen as a target platform for our research.
With the growth of platform abilities, applications become more complex to satisfy the increasing user needs (Crittercism, 2014).The increased complexity means that there are many aspects that should be taken into consideration when testing functional suitability, performance efficiency, compatibility, reliability, maintainability, and portability of iOS native business applications.
Enterprise workers are always more interested in information security than private users.The level of security is one of the factors why iOS has a dominant position in enterprise market (Eston, 2012), especially in Bring Your Own Device (BYOD) market space.While the operating system itself provides capabilities for secure application creation, they often are neglected in favor of time to market rush.That is why testing of functional security is a hot topic as well.
In order to reduce the time needed for regression testing, to spare more time for exploratory testing, or just to decrease the costs, tests to be automated.Tests can be automated at various levels.In terms of return on investments including the maintenance costs, the following test coverage model is thought to be the right one in the ideal world: most of the tests are automated at the unit level; the least of the tests are automated at the UI level; different types of integration tests lay somewhere in between.The session based/ exploratory manual testing ensures confidence in automated tests.The model is depicted in Fig. 1.1.(Scott, 2012).(Scott, 2012) While according to this model, tests at the UI level have the least coverage, these automated end-to-end tests are still very important to ensure the general confidence that the previously developed app functionality, as well as the basic UI interactions are still up and running.Automated tests from this level are probably even more important for mobile apps, because there are many gestures like tap, double tap, swipe, drag, etc. to be checked.
Several solutions have already been created or adapted for mobile UI test automation, in particular, for iOS apps.However, they all have their pros and cons, while there is no study that exposes them to choose and adapt the right one for the environmental context of a testing organization.There are also no studies that investigate the capabilities and limitations of OEM Apple UIAutomation tool.
All this increases the need for the multi-edge iOS applications testing approach that extends the systemized knowledge in the mobile testing field in general.Solutions that overcome part of the limitations of a native automator like changing the connectivity, assertions based on image comparison, advanced UI element search, repeatable executor for checking the wait conditions, simulation of memory warnings, etc. are united under tTap extension developed by the authors.

Inventory of Testing Ideas
The authors have performed the inventory of testing ideas (Kuļešovs et al., 2013).It was inspired by Kalninsh and Borzovs (1981).This activity resulted into the ideas division among the following eight classes: • Fundamental ideas.
• How to detect the correctness of the test result?
• How to detect the completeness of the testing?
• What to test (object)?
• Which quality attribute (characteristic) to test?
• Unclassified.Three millennial fundamental testing ideas are: • Errare humanum est -To err is human.
• Aliena vitia in oculis habemus, a tergo nostra sunt -The vices of others we have in the eyes, in the rear of our own.
• In propria causa nemo judex -No one can be judge in his own cause.Other testing ideas were identified through analyzing the testing terms from ISTQB Glossary (ISTQB, 2012).As a result, a map showing the linkage between the testing terms and their relation to the definite class was generated (see http://science.df.lu.lv/kaab13).It was produced using the tool that adopts the term graph building algorithm developed by Arnicans, Romans, and Straujums (Arnicans et al., 2013), (Arnicans and Straujums, 2012).

Software Testing Review on Meta-level
From practical point of view software testing mainly can be expressed by testing strategy and testing tactics on the meta-level (i.e. on the higher level of abstraction).Contexts of real testing project and theoretical background and experience of testing team influence the selection of the strategy and/ or tactics and the usage of principles of testing schools in the current testing project or campaign.A software testing review on meta-level is depicted in Fig. 2.1.

Fig. 2.1. Software testing review on meta-level.
Static context influences very much the testing vision and testing mission.Static context depends on the type of the organization (i.e.governmental, outsourcer, start-up, etc.) and on the type of the software produced (enterprise software, commercial software, web page, etc.).The options mentioned above are generally static during the whole product lifecycle.Testing vision denotes the aims that the testing team wants to achieve by testing.In some cases testing vision can be focused on producing the software with all high and critical software failures discovered and fixed, and 95% of medium severity failures identified.In some cases it can be to receive an acceptance sign-off of the product from the customer.Testing mission denotes actions that testing team does in order to achieve the testing vision.For example, the team can use only scripted testing, or it can use the benefits of the exploratory testing as well, to receive an acceptance sign-off of the product from the customer.Or testers prepare automated tests before the development to keep the product always deliverable to the customer as testdriven development suggests.Testing schools are theoretical frameworks that define testing vision and testing mission based on the static context.
All aspects of testing schools (it can also be the mix of aspects from different schools) that prevail within the organization and are common for the definite product type influence the testing strategy of the given software project.Testing strategy describes a general approach for testing.Testing strategy consists of the specification of the roles and responsibilities of each person involved in testing, testing levels, environment requirements, overall testing schedule, testing tools, risks and its mitigations, testing priorities, testing status reporting, etc.
Testing oracles that define testing exit-criteria and those that are used as the source of the derivation of test cases and expected results (i.e.correctness oracles) should be chosen within the testing strategy definition.The selection of quality characteristics to be covered by testing process should occur during the definition of testing strategy as well.Test results completeness oracles can be defined when selecting testing tactics, because often there are much more details about expected results amount available during tactics selection process.
Dynamic context depends on the project phase and influences the choice of the testing tactics that are appropriate for the given time frame, for the definite object under test, and for the current micro testing goal.Examples of the dynamic context factors are fulfillment of test entry criteria in time, availability of shared testing resources, the stabilization and bug fixing phase of the development process, etc. Testing tactics should be consistent with the testing strategy.
Testing tactics for each object under test are depicted in the test plan.Test plan consists of organizational and technical aspects.Testing tactic also influences the choice of the testing approach to be used to fulfill the current micro testing goals.Thus, technical aspects of the test plan should include the selection of the appropriate testing approaches, methods, and techniques.Testing artifacts (like test cases, test suites, traceability matrix, test data, etc.) to be produced by the testing process should be mentioned in the test plan as well.It is worth noting that some schools do not require formal and written test plans as a mandatory artifact of testing process.

Testing Schools
Testing society distinguishes five testing schools (Pettichord, 2009).They are: • Analytic School; • Standard School; • Quality School; • Context-Driven School; • Agile School.The schools are frameworks for categorization of test engineers' believes about testing and are their guide on the testing process.Testing schools are not competitive; they can be used in the collaborative mode as well.They all have exemplar techniques or paradigms, but they are not limited to them.
During the last decade the agile school starts taking the dominant position.The paradigm of this school is that testing process proves that system under test or its particular functionality work as expected, while the focus of other schools is to find the errors.Test-driven development is one of the agile testing school paradigms, thus automated acceptance tests are demonstrative exemplar of the school.

Systematization of Testing Terms: Approach, Method, and Technique
Despite the attempts of standardization of testing terms and ideas by different authorities, such as ISTQB and IEEE, there is still a little chaos prevailing in the testing literature, and between the testers themselves on the explicit usage and definition of the terms.
The connection and clear border between testing approach, testing method, and testing technique are not defined in the testing theory.For example, Beizer (1995) defines test technique as a systematic method: "A test strategy or test technique is a systematic method used to select and/or generate tests to be included in a test suite."In the same time, he uses test technique and test method as completely equal statements: "… here I present you with ready-made equivalence class partitioning methods (or test techniques) …" (Beizer, 1995); "[T]est execution technique: The method used to perform the actual test execution, either manual or automated" (ISTQB, 2012).Other authors, such as Kaner et al. (1999Kaner et al. ( , 2001)), Pressman (2005), and Sommerville (2007) have a mix of using words technique, method, approach, and strategy in regard to testing as well.
The attempts of making a distinction between approach, method, and technique were already performed by language teaching specialists in 1963, 12 years before the first theoretic foundation of testing by Goodenough and Gerhart was published.In 1963 Anthony provided "much needed coherence to the conception and representation of elements that constitute language teaching:" (as cited in Kumaravadivelu (2006)) An approach is "a set of correlative assumptions dealing with the nature of language and the nature of language teaching and learning.It describes the nature of the subject matter to be taught.It states a point of view, a philosophy, an article faith…" A method is "an overall plan for the orderly presentation of language material, no part of which contradicts, and all of which is based on the selected approach.An approach is axiomatic, a method is procedural".
A technique is described as "a particular trick, stratagem, or contrivance used to accomplish an immediate objective".
"The arrangement is hierarchical.The organizational key is that techniques carry out a method which is consistent with an approach." In 1982 Richards and Rogers (as cited in Kumaravadivelu (2006)) performed an attempt to enhance the framework developed by Anthony through dividing language teaching process into approach, design, and procedure.But, despite rather vague definition of terms approach, method, and technique, and not considering in any way of complex connections between them, exactly these terms are in favor of the most current teacher training manuals.(Hall, 2011) We suggest systemizing testing approach, testing method, and testing technique in the same hierarchical way, using the experience and keeping in mind the mistakes of language teaching specialist.Schematic relation between terms mentioned above is shown in Fig. 2.2.

Fig. 2.2. Relation between approach, method, and technique.
Testing approach "states a point of view, a philosophy, an article faith" that a test engineer takes when designing test cases.Testing method is "an overall plan for the orderly presentation" of testing techniques.Testing technique is "a particular trick, stratagem, or contrivance" to design a test case.
Testing techniques are united under testing methods based on the test case design formality (for black-box testing approach) or based on other common pronounced attributes (for white-box testing approach).
The "organizational key" stays the same as suggested by Anthony -"techniques carry out a method which is consistent with an approach".

Black-box Testing
Black-box is a software testing approach when test engineer designs test cases as if she does not know anything about the internal structure of the software under test.
Black-box testing approach consists of seven testing methods that are differentiated based on the source used for test case design process and based on the level of formality of test case designs.The relation between black-box testing methods and techniques is shown in Fig. 2.3.

White-box Testing
White-box is a software testing approach when test engineer designs test cases based on the internal structure of the software under test.There are three most known white box testing methods: control flow testing, data flow testing, and mutation testing.The relation between the white-box testing methods and techniques is shown in Fig. 2.4.

In-Operational Testing
Classical black-box and white-box testing approaches are mainly used applied during software development.Nowadays the new in-operational testing approach emerges.It means that validation occurs during the system run-time.Execution environment testing (Rauhvarger and Bicevskis, 2008), self-testing (Diebelis and Bicevskis, 2008), runtime verification of business process execution (Oditis and Bicevskis, 2015a,b) are the methods united under this approach.These testing methods are part of so called smart technologies idea that software should behave as a living being and adapt to, optimize in and defend itself again changing environment (Bicevska et al., 2015;Bicevskis et al., 2016).The relation between the in-operational testing methods and techniques is shown in Fig. 2.5.

Introduction
According to the different studies (Research and Markets, 2014;Markets and Markets, 2014;Citrix, 2015;Good Technology, 2013) iOS devices hold the major market share among the corporate workers.
With the growth of platform abilities applications become more complex (Crittercism, 2014) to satisfy the increasing user needs.The increased complexity means that there are many aspects that should be taken into consideration when testing mobile applications.Mobile workers mostly use native business applications on their devices; otherwise there would not be such a dominant position of the single operating system.That is why iOS native applications are the subject of the main interest for this section and study in general.
Despite the fact that the topic being hot, there are only some academic studies (Muccini et al., 2012;Dantas et al., 2009;Gao et al., 2014) performed that systemize the generic aspects that should be taken into consideration when testing the mobile applications without specifying the platform.Other studiesby Franke et al. (2011Franke et al. ( , 2012) ) that include the clear distinction between the platforms, concentrate on some narrow topic.On the other side, there are different iOS testing checklists, mind maps, blogs, etc. available in the internet.This motivates the authors to perform the systematic literature review of academic literature in the field of mobile testing and perform the literature review of the available non-academic (or multivocal, as per Ogawa and Malen, 1991) sources in the field of iOS testing (Kulesovs, 2015).
It was decided to concentrate both reviews on aspects of manual testing of such quality characteristics as functional suitability, performance efficiency, compatibility, reliability, maintainability, and portability according to ISO (2011).Usability testing is out of scope (except parts that are closely related to or are on the border line with the quality characteristics mentioned above).
The following research question was formulated: RQ: Which aspects (i.e.features and/ or limitations) influence the testing of functional suitability, performance efficiency, compatibility, reliability, maintainability, and portability of the iOS native business applications?
The results of both reviews are merged in order to answer the research question.The systematic literature review (SLR) of the academic sources was performed in order to gain the aspects of the mobile applications testing.The multivocal literature review (MLR) was performed in order to gain the exclusive aspects of iOS applications testing.The idea to perform two types of the review to consolidate the data from different sources was taken from work by Tom et al (2013).Fig. 3.1 shows the stages of sources selection for the whole review process applied in this study.

Research Methodology
The procedure described by Kitchenham and Charters (2007) was followed in order to conduct the systematic literature review.The qualitative review approach was applied in order to include a rigor into the systematic review of multivocal literature as suggested by Ogawa and Malen (1991).They define the multivocal sources as accessible, but nonacademic writings on the topic.

Summary of Reviews
Despite the fact that the search criteria for SLR includes studies starting from 2007, the first selected study by Dantas et al. (2009) was published in 2009, but the most productive years are 2012 (five studies: Franke et al. (2012aFranke et al. ( , 2012b)), Kim (2012), Marinho and Resende (2012), and Muccini et al. ( 2012)) and 2013 (three studies: Amalfitano et al. ( 2013), Haller (2013), and Khalid ( 2013)).Two studies by Franke et al. (2011) and by Franke and Weise (2011) were published in 2011, and one study by Gao et al. (2014) was published in 2014.Two studies by Franke et al. (2012aFranke et al. ( , 2011) ) are related to narrow topic of mobile application lifecycle, one study by Khalid ( 2013) is related to user complaints about iOS applications, and other nine sources by Dantas et al. (2009), Franke et al. (2012b), Kim (2012), Marinho and Resende (2012) 2013) is a white paper, and one source by Neglected Potential ( 2013) is a mind map.All the blog posts describe the testing only of one or some aspects, while other sources try to cover the whole iOS testing field.

Aspects of iOS Applications Testing
The aspects that influence the testing of iOS applications gathered through SLR and MLR are shown in Table 3.1.If a source is referred in the table before the details of an aspect, it means that aspect is just mentioned in the source without pointing the details that are related to iOS applications testing.

Operating System OS Variety
OS upgrade.

Restrictions and Privacy Settings
Safari, Camera, Siri, IAP (in-app purchase), Location Services, Contacts, Calendars, Photos, Social Networking, Microphone, Motion Activities, Cellular Data Use, Background App Refresh.

Resources Limitations
Lack of storage, amount of memory, running out of battery, processing capabilities.

Consumption
Memory consumption, battery consumption.

Connectivity Network Types
WiFi, Cellular networks, Bluetooth, Airplane mode.

Network Conditions
Strong/ no/ poor connection; connection loss.Ask for connection.

Internalization Region Formats
Date format, hour format.

Date/ Time Settings
Switching between time zones, system time too fast/ too slow.
It is claimed in many sources like in Kim (2012), Muccini et al. (2012), Haller (2013), Gao et al. (2014), uTest (2013), Neglected Potential (2013), and Addey (2013) that the variety of operating systems (OS) is an important testing aspect, while OS upgrade is mentioned explicitly only in Haller (2013).It is possible to set the restrictions on the usage of different hardware or OEM software completely or for the specific application within the iOS (App Quality Alliance, 2013), (TestElf, 2013), (Neglected Potential, 2013).
TestElf ( 2013) warns about the need to check an extended (Asian) on-screen keyboard, while Dantas et al. (2009) mentions on-screen keyboard as a generic aspect that should be taken into consideration.According to App Quality Alliance (2013) and Neglected Potential (2013) data can be shared via email or Bluetooth, or another network between the applications.According to Neglected Potential (2013) and Addey (2013) it is necessary to check application's logging and analytics features.Testing of In-App Purchase component is mentioned in Neglected Potential (2013).Testing of Web View component is mentioned both in Gao et al. (2014) andNeglected Potential (2013).
An application can be manipulated with a variety of gestures (Gao et al., 2014), (Neglected Potential, 2013).When animated transitions occur, they must run smoothly (Khalid, 2013), (Neglected Potential, 2013) irrespectively of the task executed in parallel.Testing for half pixels glitches and testing of Pull to Refresh feature are mentioned in Neglected Potential (2013).The necessity of checking the application both in portrait and landscape is noticed in Dantas et al. (2009), Franke et al. (2012b), Neglected Potential (2013), and Addey (2013).The importance of localization testing is mentioned in Haller (2013) and Addey (2013).App Quality Alliance (2013) identifies the need for testing of native characters and special symbols.It should also be checked that application works as designed when accessibility features of OS are enabled (Pound, 2013), (uTest, 2013), (Neglected Potential, 2013), (Addey, 2013), (Land, 2012).

Conclusions
To conclude, straight functional testing of mobile apps cannot differ from testing of web or desktop applications.The difference mainly occurs in the aspects related to the environmental multeity.
Variety of operation systems, hardware, operating system versions (often completely rebuilt from scratch) and modifications, screen sizes, screen resolutions and densities, browsers and their versions, makes mobile as a separate universe.This makes test team to select the most covering representatives from the universe to perform testing on.In many cases there are more representatives to be selected for cross-platform mobile app testing then it is needed for desktop or web application testing.
Mobile apps usually combine desktop and web applications behavior because they need to be available both online and offline.They also have more lifecycle states to verify in comparison to desktop or web applications can have.Online nature of many mobile apps implies checking their behavior under various network conditions.
Quite limited storage, battery, and processing power resources of the mobile devices leads to the additional checks to be performed when testing mobile apps under resources shortage conditions.There are also additional checks to be performed in order to verify that available resources are used efficiently by the app.
The single smaller touch screen interaction mechanism of mobile devices forces to check the functional usability of mobile app screens.Improper positioning of UI elements on the screen and unexpected gesture interference can lead to inability to use the in-app features.The examples could be: elements position is linked to on-screen keyboard position while only the default keyboard position is taken into consideration during the design; in-app near the screen border swipes interfere with system near the screen border swipes that open various system popovers instead.The smaller screen size and interaction capabilities of mobile device extract mobile web app testing into separate topic in comparison to the standard web application testing.

Introduction
While iOS offers the variety of security enhancement features to be used within the apps, they are often neglected in favor of time to market rush.The security basics that should be tested by test specialist are: using of secure network protocols, encryption of data base, and denying the access to application data when device is locked with passcode.One of the advanced functional security testing items is a checking of the development settings file (plist) entries in production app version.(Kulešovs, 2017)

Usage of Secure Network Protocols
Usage of secure network protocols (HTTPS -HTTP over SSL or HTTP over TLS, etc.) can be ensured by intercepting the network traffic with the apps like Charles1 , Fiddler2 , etc.It is also possible to see the encrypted content exchange as a plain data between the mobile app and the backend with such kind of tools, i.e. to perform manual integration testing.

Data Base Encryption
This is quite a basic, but very important task for test specialist to ensure that data base is encrypted in productive app version, because it is often kept unencrypted for testing purposes during development.It is possible to download the application data base directly through Xcode3 , or using such third party apps like iFunBox4 or iPhoneExplorer 5 .Then it is verified if encrypted or not by opening data base by any SQLite data base viewer.This also allows verifying some functional corner cases or data base corruption cases, because data base can be changed and uploaded back to iPad using the same stack of tools.

Locking the Application Data
iOS allows locking the access to application data on the device locked with passcode.However, this should be managed by app itself.That is why this feature should be often rechecked, because data to lock should be explicitly defined.During new functionality development this part is often forgotten.In-memory decryption could be another possible requirement for the high-risk apps, otherwise, currently used data is being kept in unencrypted way in Cache folder.Even if there is a code block that tries to remove all unencrypted data after usage, there is a chance that it will be left unencrypted upon app crash or app removing from memory if another app needs more memory during its execution.

Advanced Functional Security Testing
One of the advanced functional security testing items is a checking of the existence of development settings file entries (NSUserDefaults) in production app version.The development settings entries could be: skipping login, using unencrypted database, choosing the advanced subscription, usage of a feature that should be bought using inapp purchase, unhiding the features currently under development, advanced debugging, and all other staff that needs some extensive interaction with app to be achieved.There are different cases how attackers can learn about those development features: • Development and test settings files could be left together with production settings file, but not used (see Fig. 4.1).
• Entry points for development and test settings are left accessible within run-time of production version.This means that the first action item should be the adjusting of build process to leave only production settings file for the production build.The second action item should be, respectively, profiling the application code to disable the entry points for development and test settings for production build.
If not, then app settings file could be accessed using the apps mentioned before (Xcode, iFunBox, iPhoneExplorer), modified, i.e. appropriate development or test settings could be added, file could be uploaded back to iPad, and attacker could enjoy the benefits.
If the first action item is a self-explanatory, then to understand the severity of the second action item some more information on how to break the app without accessing the development and test settings files should be given.
It is possible to get the run-time properties of the app on the jail-broken device using such apps available on Cydia App Store like Cycript6 , iNalyzer7 , etc.These properties are shown in key-value format, even if they are not set from the current settings file.Then attacker just adds the desired settings and their values into the settings file and uploads it back to iPad to enjoy the benefits.

Discussion and Implications
While application security in most cases is tested by the security specialists, it is cheaper to verify that security mechanisms provided by OS vendor are used as much as possible (if the nature of the app needs it, of course) before giving the app to them.The suggestions given above allow decreasing the panic when app is checked for security when it is already in or close to production.From the authors' experience, it is often the case when product owners rush to release the app, while outsourced security specialist overloaded schedule does not allow performing the check before the target date.

Introduction
As already mentioned in the introductory part of the thesis, in order to reduce the time needed for the regression testing and to make more time available for the exploratory testing or just to decrease the costs tests tend to be automated.
Tests could be automated in the various levels.In terms of return on investments including the maintenance costs the test coverage model depicted in Fig. 1 is thought to be the right one in the ideal world: the most of the tests are automated on the unit level; the least of the tests are automated on the UI level; different types of the integration tests lay somewhere in between.The session based/ exploratory manual testing ensures confidence in automated tests.(Scott, 2012) While according to this model the tests on UI level have the least coverage, these automated end to end tests are still very important to give the general confidence that previously developed app functionality, as well as basic UI interactions are still up and running.Automated tests from this level are probably even more important for the mobile apps because there are many gestures like tap, double tap, swipe, drag, etc. to be checked.

Solutions for Automated UI Testing of Mobile Apps
There are several solutions already created/ adapted for mobile UI test automation, in particular, for iOS apps.The solutions could be divided into several groups based on the origin, cross-platformance, and the way of executing the automated commands.
The first big clusters are OEM automation tools vs. the third party automation tools.OEM automation tools come together with the mobile OS manufacturer IDE.All other mobile automation tools are the 3rd party solutions.The most of the solutions use API-based approach for recognizing the object on the screen, while there are some solutions that use image-based approach for the same purpose.API-based solutions can be divided into two more groups: wrappers above the native automation tools vs. others that have the prerequisite to incorporate the custom library into the app source code.Some of the solutions offer to run the tests in cloud.While almost each solution nowadays can run tests both on device and on simulator on premises, only some solutions support running the tests on the real devices in cloud.

Apple UIAutomation and XCTest
UIAutomation tests are written in JavaScript.The framework consists of the most basic functions for all UI elements available in iOS.(Apple Developer, 2012) The access to some device functions like sending app to background, changing the volume, setting the location, etc. is also available.If some custom UI View is used inside the app it can be accessed as UIAElement classthe superclass for all user interface elements in the context of the UIAutomation.
Starting from XCode 7 Apple added the possibility of writing the UI automated tests on Swift language and to run them on XCTest framework (it is a unit test style framework for Swift/ Objective-C code) inside Xcode IDE itself.In terms of the functional scope of API both UIAutomation and XCTest frameworks are on the same level.
Being the frameworks with the powerful set of basic functions, one of the issues for both of them is that the commonly used test notations from these basic functions are quite wordy.Several extensions have been created for JavaScript based UIAutomation in order to enable the ability to write the tests using the less repetitive higher level commands in a style more common for the testers.Each extension follows the notation style convenient for the creator.Both most popular extensions are distributed under MIT license.There are no extensions available for XCTest UI testing framework yet.

Tuneup JS
The main achievement of TuneupJS8 is the creation of the unit test like test runner and providing the extensive set of assertions.The extension has the image comparator inside that is based on ImageMagic9 tool.It also consists from the set of the commands that combine several UIAutomation basic commands into one higher level command making the notation shorter.mechanic.jsmechanic.js10 is a CSS-style selector engine for UIAutomation.It also allows accessing UIAElements and executing the commands with a shorter notation.

Google Testing Support Library
Google Testing Support Library consists of three main parts11 : • AndroidJUnitRunner: JUnit 4-compatible test runner for Android.
• Espresso: UI testing framework; suitable for functional UI testing within an app.
• UI Automator: UI testing framework; suitable for cross-app functional UI testing across system and installed apps.
UI Automator functionality on Android is similar to UIAutomation functionality on iOS, while Espresso could be described as white-box UI test automation tool.Testing Support Library Tests are written in Java.

Microsoft Coded UI Tests
Coded UI Tests is an analogue for UI test automation for Windows apps12 .This tool supports almost all Windows-based platforms, not only mobile ones.It could be even used for web apps UI test automation.The tests are written in C#.

3 rd Party Solutions and Summary
The market players with characteristics they posses are summarized in Table 5.1.,Table 5.2, and Table 5.3.All of the solutions have record/ play capabilities.That is why this option is excluded from the comparison tables.Each solution also supports the CI setup.
The difference between them all lays in the progression described below: • OEM automation tools are the most robust one between the API-based tools.They come with a sufficient set of functions to build the commonly used test patterns, but in case of Apple UIAutomation the scripting is too wordy.They also are limited to the one platform.
• Wrappers are cross-platform solutions.Appium tool is the only wrapper so far.The vendors of the several other tools have adopted their cloud testing labs (with real devices) to run Appium tests.Wrappers add some additional weak points per platform, per script language, per environment.It means that if something does not work then the 13 http://appium.io/ 14https://developer.xamarin.com/guides/testcloud/introduction-to-test-cloud/ 15http://www.tricentis.com/tricentis-tosca-testsuite/tosca-mobile-plus/ 16http://docs.telerik.com/teststudio/test-studio-mobile/overview 17http://www.keynote.com/solutions/testing/mobile-testing 18http://www.ranorex.com/test-automation-tools.html 19https://docs.experitest.com/display/public/SA/Manually+Instrumenting+iOS+Applications 20 http://www.sikuli.org/testing.html 21http://docs.testplant.com/ePF/using/epf-getting-started-ios-gateway.htmissue could be related exactly with the code that does wrapping, while the same command would work in the OEM automation tool.The solutions that need the 3rd party library integration into the source code have the same pros and cons as wrappers do.But there are two additional weak points: • The code of the app under tests is changed in comparison to the release version.It increases the probability of app working differently when it is built for the automated testing purposes.Of course, the same applies for all automation solutions, because they all interfere into the app under test in some way.But there is more trust that this interference is properly handled when the OEM solution is used.
• It is not possible to access the system modal windows/ popovers and device functions from these libraries.Test framework can access them only by calling the methods of OEM automation API.
Image based tools can simplify the recognition of UI object in a short term, however, having them as the only solution has the following cons: • Early automation is hardly possiblewith agile software development approach it is very possible that image slices are not yet available, while functionality is already there.
• UI can vary not only per platform, but also per device type (phone vs. tablet).In case of image based tools it will increase the test creation and maintenance costs.
• Adjusting/ refreshing UI up to new OS guidelines most probably will trigger more test maintenance effort than it would be needed for API-based solutions.
To conclude, tests written using the tools that are using image pattern recognition of the UI object are quite fragile in comparison to API-based solution, while having the image comparison for assertion in some cases is the only way to go for UI level tests.More detailed overview of the solutions can be discovered in Kulešovs (2017).

Apple UIAutomation Capabilities and Limitations
Before choosing UIAutomation as a target test automation tool more deep analysis of its capabilities and limitations was performed.The identified functional blocks were divided into several levels: application level, OS level, device level, device/ OS level combined, and framework level.(Kulesovs et al., 2015)

Application Level
On application level UIAutomation is capable to interact with all native UI elements, as well as interact with custom developed UI elements that either extend or customize the native UI elements or are totally custom designed UI elements that extend the top UI elements -UIView or UIViewController.
The tool also supports all native gestures, however the native pinch to zoom gesture does not work on simulator starting already from iOS 7. The support of custom gestures is quite limited.Custom gestures can be simulated only if they can be performed with a single drag between two points.It also is not possible to simulate the complex drag gesture between more than two points.So, if it is needed to simulate the drawing of a curve, this could be achieved only by performing a large set of drag gestures, finishing each of them as if the finger was taken out from the device screen.Even if result will look mostly the same (the circle is drawn), this is achieved with completely different internal logic.
It is possible to change the app settings (stored in setting property list file) during the test run.However, if it is needed to change the setting before the test run (e.g. the setting that is applied when the application starts), then the setting should be adjusting during the build process or app to be restarted after setting is changed during the first run.Restarting the app is possible only by stopping the current test run and by starting another one.
There is an option to simulate memory warnings when app runs on the simulator.However this function is not accessed in UIAutomation out of the box.
UIAutomation understands UIWebView structure, so it is possible to interact with the web content.However, web apps to be built in into the native app to test them using UIAutomation.The tool is not supposed to run the default Safari browser (and any other built-in app), that is why there is a very basic custom web browser app with a single web view is created by the community to test the web apps on iOS.The issue is that if web app has complex JavaScript inside then web view should also implement much more functionality than a single default web view.

OS Level
On OS level the tool is able to send the app to background for a definite amount of time.However, it is not possible to switch between the apps, even if they both are custom built.Switching between apps could be useful, for example, if there is an intention to check how Open In works for the app under test.It is not possible to manipulate with push notifications as well.

Device Level
On device level it is possible to perform the orientation change of the device.This is very important feature for mobile UI tests.There is also an ability to simulate the pressing of device buttons to change volume, lock and unlock the device (without the passcode), simulate as if it is being shaken.

Device/ OS Level
On the combined device/ OS level the tool allows to manipulate with the location services, i.e. to set the latitude, longitude, altitude, course, moving speed, etc.However, the tool is missing the support of switching on/ off the WiFi connection, as well as the ability to change the connectivity speed.The tool is also missing the ability to manipulate with the restrictions, privacy settings, and region formats.Simulating the interruptions like receiving the phone call or SMS is also not possible.
Often there is also a need to add and/ or remove the images from Photo app and contacts from Contacts app to create different preconditions for test execution or to perform the cleanup before or after the test.It is not possible to perform such actions out of the box.However, there is a possibility to execute the tasks on the host Mac machine that from which tests are executed.

Framework Level
On framework level UIAutomation is missing unit test style notations test runner.It also is not able to search for the element within the whole element tree.It searches only within the first level children out of the box.There is a set of functions to check some basic conditions like if element is present, however there are no out of the box wait statements, as well as more complex conditions are needed to check if the tool can interact with the element.
The tool is capable to take a screenshots while the test being executed.But there is no built in comparison inside it.Typing on the keyboard also fails from time to time when characters from the different keyboards are being typed in (e.g.letters and numbers and/ or special symbols, letters in capital and in narrative).

Summary
To summarize, Apple UIAutomation can perform the most of the basic functions that can be executed on the iOS device.However, there are still several limitations in terms of functionality and framework usability.The described capabilities and limitations of the tool are aggregated in Table 6.1 and Table 6.2.The main reason for this title is that almost all actions within the extension are executed in absolute coordinates of the device while still operating on the UIAElements (UIView and UIViewController) level.The device (or simulator) is called target in UIAutomation context.The decision to work in absolute coordinates was made to overcome several issues that we will describe in a course of this section.It is worth mentioning that tTap extension is distributed under MIT license24 .

Solution Details
The goal of tTap is to overcome the limitations of the original UIAutomation aggregated in Table 6.2.The details on overcoming each limitation are described below.Overcoming of limitations on the application level, OS level, and device level is achieved through the execution of AppleScripts to manipulate the built-in OSX application that can interact with an iOS device on the host machine (the machine to which iOS device is connected during the test execution).On framework level the limitations are overcome through triggering the execution of shell scripts to perform, for example, image comparison, etc.New JavaScript functions are written to extend and enhance the existing framework as well.

Application Level
The end to end support for custom developed gestures could not be achieved at all.The only option is to integrate the custom library inside the application under test that will call the same method execution as if custom gesture is performed.However, this is a big overhead, because normally there should be the way to use the same functionality using the simple tap, long press, or swipe gesture.The support for complex dragging gesture (more than two points) can also be achieved through the custom library integration into the application under test.But, again, there should be very strong argument for doing so.
It is possible to simulate low-memory warnings when app runs on simulator.This could be achieved through calling the respective function from the simulator menu (see Fig. 6.1.).To do this in automatic way tTap contains function that runs AppleScript that "clicks" this menu item using the hotkeys.

OS Level
It is not possible to switch between the apps from UIAutomation framework.The issue is that even if another app is opened by calling it through URL schema, UIAutomation does not see it.It can be attached only to one application during the test run.
Quite often apps perform some internal navigation when they are opened from push notification.In general, it is possible to manipulate with alerts from UIAutomation.However to make such test repeatable is very hard, because the notification is sent through Apple Push Notification Network in real time.That is why the time when the notification arrives to the device is unpredictable.

Device/ OS Level
There is no way to switch on/ off WiFi connection or any other connection through UIAutomation.However, it is possible to share the WiFi connection from Mac machine and use it on iOS device where tests are executed.In such case, the shared WiFi could be switched on/ off by using one of the tools from Apple Developer toolset called LinkConditioner (see Fig. 6.2).There is also an option to simulate the connection speed of different connection types like EDGE, 3G, etc., as well as to simulate the bad network conditions or to create custom network conditions (see Fig. 6.3).
AppleScript to manipulate the LinkConditioner tool from UIAutomation is a part of tTap framework.The AppleScript contains the functions to run/ close LinkConditioner, to select the definite network condition profile, and to switch the selected profile on/ off.
There are several ways how to deal with the initial set of photos and/ or contacts for the automatic test presetup.One way is to use tTap commands that run specific AppleScripts to manipulate with photos and contacts.It is possible to open such OSX apps as Photos and Contacts and use their functionality to add/ remove data from the device.This presetup is especially important in case of there are several apps being tested on the same device.Another way is to build and install the custom app that adds/ removes such kind of data from the device.
In general, it is possible to simulate the different types of interruptions like phone call, receiving of SMS, etc.However, this is not needed, because it is up to operating system, not up to the app how to deal with such kind of interruptions.Receiving of the phone call for the app is the same as switching it to background, nothing more.
It is impossible to manipulate with restrictions and privacy settings within the app.The permissions to the photos and contacts can be asked by the app only once.
Afterwards OS prevents app from asking them again, even if user uses the functionality of the app where these permissions are needed.User can change the restrictions only manually in app or OS settings after giving the response on the first prompt.These security features of OS make it impossible to automatically check the app behavior when there are some restrictions set for the app, because it limits the testing of positive cases afterwards without a manual intervention.
To have a constant automatic check of app behavior under different regional and time setting is possible only if tests run on the multiple devices with different regional and time settings being preset.

Framework Level
The modified test runner from Tuneup JS is used to run the tests.Tests follow the unit tests style convention.The test suite is wrapped into JavaScript function.There is a separate file where these "test suite" functions are called in the definite order.We have extended the test runner with the possibility to ignore the definite tests and make some tests dependant from another tests result (e.g.not to run the test if the precondition is not achieved).The advanced assertions capabilities are taken from Tuneup JS without modification.
UIAutomation allows searching for the element only within one node of the UI elements tree.tTap implements the recursive search by accessibility identifier from the root node or from the definite parent.The idea is taken from Penn (2013).
As already mentioned, almost all actions are made on device (target) level.This is the closest way how touches occur in reality.Gestures are executed on the target using the calculated center point of UIAElement in absolute coordinates.iOS recognizes the object at these coordinates and go through the responder chain searching the element that executes the actions responding to the definite gesture, as shown in Fig. 5.4.This solves the following: • There are cases when UIAutomation does gesture on the wrong coordinates if the command is called exactly from the UIAElement.It more often occurs with the system windows like email controller or some context menu, especially if app is created using some cross-platform solutions like Xamarin.We have not searched for the reason, but this workaround works perfectly.
• By default UIAutomation does tap at (0, 0) point of UIAElement, while the real user tends to tap to the center of the object in the most cases.
• This led to the idea of creation such convenient and often used function as UIAElement1.tDragAndDrop(UIAElement2)where the object on top of which to drop the current object is set as a parameter.Other than that several improvements to the framework that simplifies it or makes it more robust have been developed.They are: • unit test style notations; • advanced assertion capabilities; • searching within the whole UI elements tree; • various wait conditions; • improved keyboard typing robustness; • advanced logging/ debugging capabilities.
In order to perform a stress testing tTap also allows simulating low-memory warning, however only on simulator, not on the real device.

Ideal Cross-Platform Mobile UI Test Automation Tool Proposal
The analysis of the mobile UI test automation tools from the fifth section, the analysis of the possibilities and limitations of the out of the box Apple UIAutomation, and creating the solutions for these limitations united in tTap frameworks resulted into the proposal of the ideal cross-platform mobile UI test automation tool creation.The device control is achieved with EggPlant image-based tool instrument -custom Springboard.More matured Appium tool (that being a wrapper on top of native automators has the best concept for cross-platform support) is used for test automation itself.Its part for iOS apps automation is extended with the solutions united in tTap framework.Android UIAutomator and Microsoft Coded UI Tests capabilities and limitations still to be investigated and solved if possible.Even Appium wraps only iOS and Android native automators, there already a project called Winium26 started that wraps Coded UI Tests as well.The proposal is schematically depicted in Fig. 7.1.Using custom Springboard from EggPlant to control the device is cleaner and most efficient way in case of iOS.Jail-break is a no go for device control at all.Using wrapping concept is the cleanest way possible for cross-platform UI test automation solution.

Conclusions and Discussions
The study consists of six main parts.Each part adds value to the software testing field in general and to the field of the mobile software testing in particular.
The first part concentrates on the inventory and structuring of testing ideas and terms.It has resulted into discovering of eight classes of the testing ideas.Initiation of such process has helped to understand the need of making the clear definition of such terms as testing approach, testing method, and testing techniques that has been achieved using the solution made by Anthony in the field of language teaching.Testing methods and techniques have been united under black box, white box, and in-operational testing approaches.Structuring of the ideas have also made it possible to schematize and visualize the software testing on meta-level, defining the relation between such concepts as testing strategy, testing tactics, testing schools, testing mission, testing vision, different (organizational and project-wide) contexts, testing approach, testing method, testing technique, testing plan, etc. Uniting various software testing processes under software testing tactic term has been made for the first time.The visualization of the relation and clustering of the software testing ideas and terms has been done for the first time as well.
In the second part the aspects of mobile applications functional testing are investigated.iOS was chosen as a target platform for investigation, because it is the current market leader in the enterprise world.The literature review of both academic and multivocal literature was performed.The majority of the sources selected for the review, both academic and multivocal, were published during the last seven years period.
The results of SLR are mostly related to general mobile applications testing aspects like limited resource utilization, orientations, localizations, etc., while the results of MLR provided the needed details of iOS application testing aspects (like definite restrictions and privacy settings, iOS accessibility features, etc.), as well as identified some new aspects like IAP, date/ time settings, etc.The identified aspects were divided between 4 large clusters: Environment, Application Lifecycle, Inside the Application, and (functional or performance aspects of) UI/ UX.
The third part looks into the functional security testing of iOS applications.While application security in most cases is tested by the security specialists, it is possible and is much cheaper to verify that security mechanisms provided by OS vendor are used as much as possible (if the nature of the app needs it, of course) before giving the app to them.This often is neglected in favor of time to market rush.These mechanisms are usage of the secure network protocols, data base encryption, and locking the application data.Another functional security testing part is to check and eliminate the leftovers of development and testing activities in the productive build of the app.The examples of leftovers are settings files and code that reads them or performs the action based on the setting value.This issue is not discussed in the literature at all.The authors got the details through the own studies while breaking the apps.
The solutions for mobile UI test automation are discovered and categorized in the fourth part.These solutions can be divided into three parts: OEM tools, API-based tools, and image comparison based tools.API-based tools can also be divided into two groups: wrappers (tools that wrap the native automators) and tools that need 3rd party library integration into the application code.All non OEM tools are cross-platform tools.
Tests written using the tools that are using image pattern recognition of the UI object are quite fragile in comparison to API-based solution, while having the image comparison for assertion in some cases is the only way to go for UI level tests.
The study of the (mobile) automation tools available on the market has been performed by practitioners many times, for sure.However, most of them are not publically available, but those who are do not provide any thorough categorization of the tools.
The rigorous study of the capabilities and limitation of Apple UI Automation has been done for the first time.This is reflected in the sixth section.These capabilities are divided among several levels: application, OS, device, and OS/ device.UIAutomation capabilities are: interact will all built-in UI elements; interact with custom developed UI elements; support for all built-in gestures; web-views; changing app settings; sending app to background/ foreground; simulating device buttons pressing (i.e.volume, etc.); orientation change; manipulation with location services.Out of the box UIAutomation limitations are: support for custom developed gestures; support for complex dragging gesture (more than two points); low-memory warnings, switching between apps; open app from push notification; switching on/ off WiFi connection; changing the connectivity speed; restrictions and privacy settings; region formats; interruptions; add/ remove images from Photos app; add/ remove contacts from Contacts app; unit test style notation; limited assertions capabilities; searching within the whole UI elements tree; image comparison; wait conditions; robustness of keyboard typing; limited logging/ debugging capabilities.
All limitations were analyzed and solutions were provided for those that do not require to jailbreak device or perform any other hacking of the OS or device.These solutions are united in tTap frameworkthe extension for Apple UIAutomation.The following limitations are solved in tTap framework: low-memory warnings (on simulator only); switching on/ off WiFi connection; changing the connectivity speed; add/ remove images from Photos app; add/ remove contacts from Contacts app; unit test style notation; limited assertions capabilities; searching within the whole UI elements tree; image comparison; wait conditions; robustness of keyboard typing; limited logging/ debugging capabilities.
The investigations, analysis, and tTap solution creation from the fifth and sixth sections led to the ideal cross-platform mobile UI test automation tool proposal.The device control is achieved with EggPlant image-based tool instrumentcustom Springboard.Appium tool (together with Winium spin off for Windows) as a wrapper concept is used for test automation itself.Its part for iOS apps automation is extended with the solutions united in tTap framework.This is also the fact, that the mobile applications field is not mature enough yet.It is even less mature in terms of testing.The authors investigations on the iOS apps testing aspects, iOS apps functional security aspects, and mobile UI test automation, including the creation of tTap framework and making the proposal of the ideal cross-platform mobile UI test automation tool, focuses on this issue and makes the field a bit more mature.

Fig. 4 . 1 .
Fig. 4.1.Example of the development and test settings files in production build.

Table 3 .1
Aspects of iOS Applications Testing Screen size, resolution & pixel ratio, pro-cesssing efficiency, memory, storage capacity

Application Update Inside the Application Keyboard
Extended keyboard.Data Import/ ExportEmail; Bluetooth/ network (peer to peer).

Table 5 .
1. OEM Solutions for Mobile UI Test Automation

Table 5 .
3. Cross-platform Solutions for Mobile UI Test Automation -Characteristics

Table 6 .
1.The Capabilities of Out of the Box Apple UIAutomation Capabilities

Table 6 .
2. The Limitations of Out of the Box Apple UIAutomation LimitationsTuneup JS extension that resulted into new extension creation that we call tTap 23target tap.

Table 6 .
3. The Status of Overcoming Apple UIAutomation Limitations