The ever increasing demand of mobile devices has given a push to software developers in taking the traditional web applications to mobile environment. The challenge is to provide user experience as similar and seamless across various mobile devices as possible in spite of the limitations which the mobile environment poses, adopting an agile methodology to develop the mobile applications for a diversified device environment, hardware and networking considerations.
Mobile device markets that includes Smartphones, Tablets, PDAs etc. is growing dynamically making the mobile application developers strive to deliver most robust, scalable applications with quality assurance Every device platform creates a unique testing environment challenging the mobile application developers to follow different testing strategies. Here we shall see how different types of testing approaches can be taken up for a variety of mobile platforms.
Numerous different mobile platforms available for mobile applications to name a few:
- Apple’s iOS
- Google’s Android
- Nokia’s Symbian, Maemo and MeeGo
- Palm/HP’s WebOS
- Samsung’s Bada
- RIM’s BlackBerry OS., and many more.
As mentioned, every platform needs a different testing approach. A combination of manual and automation testing can be done for an effective outcome.
Following are different types of manual testing for mobile environment:
- Functionality Testing:
Functional testing mainly includes finding device specific bugs and navigational issues of application. This type of testing should be done at the initial stage when the application is under development. In functional testing we can check database or network queries with response times, crashes and memory issues. Functionality testing of a mobile application is a black-box type of testing to assure that the application is functioning as per the business specifications.
- Usability Testing:
Usability testing encompasses mobile interface testing, application navigation testing, and intuitiveness of the application, consistency, and soberness of color scheme. An ISO standard defines usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use”.
In Usability testing ease and efficiency of the user and content verification will be tested as defined in common Usability guidelines.
- Network Connection Testing:
Every mobile application which utilizes internet needs to be tested under different networks. These are a few networks for testing applications:
2G (GPRS, CDMA, EDGE), 3G and WiFi.
This is a necessity because most of the times the server responses are different for different type of networks. This testing helps us check application behavior under different networks.
- Installation Testing:
While installing, the application should install required softwares automatically on the device. And while uninstalling the application, it should remove all the available content and databases from the device which are used by application.
- Performance Testing:
Performance testing is required for finding memory related issues in application when data is excessive, due to devices having less memory as compared to simulator, this testing is required to be done on device only. By this testing approach, we can find the stability and performance of application under excessive data.
- Stress Testing:
This testing is required for getting the application response time while clicking on different buttons regressively and adding more data on device. We can get different crashes and hangings of application while running the application for long time.
In absence of mobile devices, simulators have always played a vital role for testing mobile device applications, although device testing is always the ideal way as it represents more likely end user scenarios, the significance of simulators in testing cannot be ignored. To have an effective testing with the help of a simulator, it’s necessary to explore all the capabilities of simulator.
Simulator versus device testing:
Simulators are mainly used for functional testing of basic flows. Simulators are not used for performance and usability testing, but the final testing is conducted on actual devices so crashes and hangings can be identified. One cannot get quality application while relying only on simulator. Device testing is necessary for all the applications, as with device testing we can understand the application behavior on different networks.
Simulator:
Is a software application that can accurately imitate a mobile phone. Simulator is mainly used by developers to check the functionalities implemented under development phase.
Let’s discuss the pros and cons of using simulators:
- Pros:
- Helps in isolating issues which are not volatile network connection dependent
- Provides a wide variety of testing over different types of device simulators for the same build
- Allows to test the same build in multiple device screens.
- Cons:
- Simulators of older generation handsets don’t resemble the device as closely
- Some issues which are hit by the speed at which input was given cannot be reproduced easily
- Hardware/Firmware environment variations detectable in device testing only
- Device testing is always preferred as it represents more likely end user scenarios.
Device: Is the actual handset where application installed and runs.
There are some pros and cons while using real devices for testing as well.
- Pros:
- Finds actual issues of application.
- Finds crashes, memory leak issues which can not found on simulators.
- Checks application over 2G and 3G and different networks
- Checks application behavior while incoming call, SMS, MMS and alarm.
- Cons:
- Expensive for compatibility testing of application over wide range of devices
- Consumes more time for adding excessive test data for testing purpose.
Checklist to follow while testing a mobile application:
Following is a basic checklist which is required while testing mobile application for any platform:
1. Installation & Uninstallation: To verify whether the application can be installed & uninstalled successfully.
2. Network Connectivity:
- The application can use simultaneous connections properly
- The application follows the GSM Offline profile correctly when making connections.
- When GSM Offline profile is selected, application cannot take network connection or send an SMS/MMS
- The application can utilize WLAN, 2G and 3G networks correctly.
- Performance of application during network connectivity problem.
3. Call/SMS/alarm handling: Verify that Application pauses and resumes for the same state when there is an incoming phone call/SMS/MMS/Alarm notification.
4. Check the look & feel of the application
5. Content: Check if enough information is displayed
6. The application must function as defined in the Help, user’s guide, or functional specification
7. Performance : Application and inner pages load time.
Automation Testing:
Automation testing is usually extended to perform tasks impossible with manual testing in large applications. An automated software testing tool is capable of playing pre-recorded and predefined actions. The results are mapped to the expected behavior and report the success or failure of these manual tests. Once automated tests are created they can easily be repeated. Having experienced the efficiency of automated software testing, it has become an important part of an application testing.
Automation tools available for testing:
Mainly mobile testing is done manually on actual devices. Some of the following tools are available in to test the functionality as well as usability of application.
- Robotium for Android
- Testquest, try, and digia for Symbian
- FoneMonkey for IPhones
- Memory sweep for IPhone
- Other tools: eggplant, VNC Robot, Hopper and TestQuest.
Advantages and Limitations of Automation Tools:
Advantages:
- Multiple tests can run in less time with fewer resources
- Automated Tools run tests faster than human users
- Can reuse tests on different versions of an application.
Limitations:
- Unable to test all the functionalities and Usability of application through automation tools
- Proficiency is required to write the automation test scripts for application different functionalities
- Debugging the test script if any error is present in the automation test script
- Storing and maintenance of test data is difficult.
I believe that this comprehensive information will be quite helpful to you if you are looking at some methods for an effective approach to test a mobile device application. You can leave your comments and/or questions; I would be looking forward to having an interactive conversation.